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Abstract 


This report presents preliminary findings from a cognitive task analysis 
( CTA ) of business aviation piloting. Results describe challenging 
weather-related aviation decisions and the information and cues used to 
support these decisions. Further , these results demonstrate the role of 
expertise in business aviation decision-making in weather flying, and 
how weather information is acquired and assessed for reliability. The 
challenging weather scenarios and novice errors identified in the results 
provide the basis for experimental scenarios and dependent measures to 
be used in fixture flight simulation evaluations of candidate aviation 
weather information systems. Finally, we analyzed these preliminary 
results to recommend design and training interventions to improve 
business aviation decision-making with weather information. The 
primary objective of this report is to present these preliminary findings 
and to document the extended CTA methodology used to elicit and 
represent expert business aviator decision-making with weather 
information. These preliminary findings will be augmented with results 
from additional subjects using this methodology. A summary of the 
complete results, absent the detailed treatment of methodology provided 
in this report, will be documented in a separate publication. 
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1.0 Introduction 


On February 12 of 1997, President Clinton called for a five-fold reduction in the rate of fatal aviation 
accidents within a decade, and directed NASA to support this national safety goal. Recognizing that 
weather contributes to approximately thirty percent of aviation accidents, Aeronautics Safety Investment 
Strategy Team (ASIST) workshops suggested that a NASA program should focus on improving the 
quality of National Airspace System (NAS) users' weather information (NASA 1997). NASA created the 
Aviation Safety program to accomplish this presidential aviation safety initiative. This program includes 
the Weather Accident Prevention project, which addresses a variety of the weather-related safety 
initiatives suggested by ASIST workshops. The objective of the constituent Aviation Weather 
Information (AWBM) element is, specifically, to improve pilots' weather situation awareness and the 
quality of weather-related flight decisions. This research project supports AWIN's objective by 
investigating decision-making of business aviators in weather flying. 

1.1 Weather Flying 

It is not surprising that weather is a major contributor to aviation accidents. Weather represents perhaps 
the most dynamic and least predictable aspect of the NAS environment. The dynamics of weather 
phenomena are multi-dimensional. Its characteristics vary in three-dimensional space, over time, and in 
intensity. Pilots' views on weather phenomena are limited by the weather information products that are 
available on the ground, and dramatically limited by the far fewer available once airborne. A pilot's 
weather situation awareness is further constrained by the limited usability of available information ( e.g ., 
coded METAR text). The reliability of forecasted weather information is diminished by the 
unpredictability of the phenomena. Both forecasted and immediate weather information are only as 
reliable as the sensors that obtain and, in some cases, algorithms that interpret raw atmospheric data. 

Thus, even perfect information, perfectly perceived does not guarantee safe flight in difficult weather 
conditions. Finally, as aviation accidents and incidents result from a confluence of events, even perfect 
weather information that is understood perfectly must be considered in light of its implications for a 
particular flight. Weather information is only one aspect of the multifaceted decisions pilots must make. 
Pilots must integrate their understanding of weather conditions with understanding of the terrain, aircraft 
performance characteristics, airport facilities, their own skills and capabilities, airspace constraints, etc. to 
arrive at safe decisions in situ. Underlying these concerns is the normal multitasking nature of aviating, 
and the commensurate undulation of pilot workload. Pilots' susceptibility to both overload and 
complacency therefore further exacerbates problems of weather information interpretation, and aviation 
decision-making with weather information. This investigation focuses on the weather flying knowledge, 
skills and decisions of business jet pilots to better understand how to improve, and improve the use of, 
aviation weather information. 

1.2 Business Aviation Operations 

Business aviation has an excellent safety record. Since the mid-1980's, accident rates among 
corporate/executive operators have been lower than those of other operators in FAR part 135 operations 
and of any segment in general aviation, and comparable to FAR part 121 operators (NBAA 2000a). 
Despite this exemplary safety record, AWIN, as well as other NASA Langley efforts (Schutte & Wiltshire 
1996), have recognized that business aviation offers several advantages for developing and implementing 
advanced flight deck concepts. The variety of business jet operational contexts serves as a broad basis 
from which to extend technologies to both general aviation and transport aviation. This breadth of 
operational contexts presents many of the most challenging conditions for flying. 


1 



There are also implementation advantages. Business aviation owners and operators are more likely to, 
and quicker to, adopt new technologies than those of other general aviation or commercial transport 
aircraft (Kauffmann & Pothanun 2000). The design cycle of business aviation jet aircraft is much shorter 
than that of commercial transport jets (Perry 1999). Business jets are redesigned more often than are 
other general aviation airplanes because their users demand new technology. These design cycle 
considerations allow more frequent opportunities for, and less resistance to, introducing new 
technologies. Finally, the market for business aircraft is increasing dramatically. In a poll of U.S. 
turbine-powered aircraft operators, use of business aircraft increased 44% in the past 12 months, and 32% 
of firms using business aircraft expected employees to make increasing use of these aircraft (NBAA 
2000b). Business aircraft manufacturers anticipate this trend to continue (Phillips 2000). This study 
focuses on business aviation for reasons of operational variety, and market penetration. The following 
sections summarize characteristics of business aviation operations according to features originally 
described in Rogers et al. (1998). 

Mission Characteristics 

Typically, business jet pilots do not fly for compensation directly, but as an employee in the service of, or 
as contracted by, a professional organization. Because the goal of business jet operations is to serve 
business needs, trips are scheduled with relatively short notice. Most trips are scheduled with about two 
weeks notice, but more urgent trips also occasionally occur. Trip destinations are also dictated by the 
needs of business. As a consequence, and as allowed by aircraft characteristics, business jet pilots fly into 
a wider variety of airports and airfields than do either smaller general aviation or commercial transport 
pilots. Frequently, business aviation missions use smaller airfields to minimize costs and distance to 
business destinations. These smaller airfields are less likely to provide complete weather information 
available at larger airports. Perhaps as a result of mission differences, business aviators tend to discount 
preflight weather information relative to weather information and observations available in flight (Lapis 
1998). Business jet pilots not only fly to a variety of destinations, as dictated by their passengers, they 
also may need to make in-flight deviations to serve the needs and desires of their passengers. 

Aircraft Characteristics 

The National Business Aviation Association (NBAA) compiles a description of the aircraft fleet used by 
business aviation pilots. Table 2 presents the percent of the NBAA fleet for each type of aircraft (NBAA 
2000c). 


Table L Aircraft types in the NBAA fleet. 


Aircraft Type 

Percent of NBAA Fleet 

Light Jets (< 29,9991bs) 

35% 

Piston-powered (reciprocating) 

20% 

Heavy Jets (> 30,0001bs) 

16% 

Light Turbo-props (<12,5001bs) 

11% 

Heavy Turbo-props(>12,4991bs) 

10% 

Helicopters (most) (< 12,5001bs) 

7% 


In addition to the basic weather information services available to all aviators, business jet aircraft usually 
have on-board weather radar systems, and may have ACARS-like capability to uplink textual weather 
information and ground services to provide additional weather information (Lapis 1998). Business 
aviation aircraft are equipped to serve the business needs of passengers, and typically include: computer 
docking stations, fax machines, air phones, and other office technology. 
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Organizational Characteristics 

The organizations that business aviators serve vary widely, and include commercial/industrial, 
government, academic, and not-for-profit agencies. Essentially, the worth of business aviation is 
established by the opportunity cost of not being able to make a trip when and where it is advantageous for 
that organization, and the opportunity cost of personnel work time absorbed by less efficient travel. 
Business airplanes that serve these needs can also vary in the infrastructure supporting operations (Rogers 
et al. 1998). Small companies, or those that share aircraft, typically fly smaller aircraft and use Fixed- 
Base Operators (FBOs) to conduct maintenance. Farger corporations, celebrity pilots, and fractional 
ownership businesses tend to use larger aircraft, and typically have full maintenance, flight operations, 
and ground support infrastructure. Farge, global companies with multiple aircraft type fleets, 
international flights, and heavy aircraft most resemble commercial carrier airlines, and have similar 
standards for dedicated training, maintenance, and operational control departments. 

Pilot Characteristics and Roles 

Business pilot experience generally increases with the size of the business' aviation operations and 
duration of missions (Rogers et al. 1998). A small survey of business aviation owner/operators, found 
that approximately half have college degrees, and most are type rated in more than one aircraft (Rogers et 
al. 1998). Training is usually governed by internal operating procedures, unless operating under FAR 
Part 135. Business aviators typically are very familiar with the particular airplane they fly. This contrasts 
with commercial transport pilots, or rental general aviation pilots who may fly different instances of the 
same airplane, or even different airplanes. However turnover rate is high among business jet pilots, as 
they graduate to larger aviation operations. While most business aircraft are certified for single pilot 
operations, they are typically flown with pilot and co-pilot. Due to high turnover and use of "rental co- 
pilots," it is not unusual for crewmembers to be unfamiliar with each other’s skills, predilections, or 
experiences; as well as unfamiliar with the business' operating and crew resource management 
procedures. 

Business aviation pilots must do more than simply pilot the aircraft to the destination. Business aviation 
pilots are particularly conscious of comfort of their passengers, their ability to conduct business during a 
trip, and their ability to make meeting times. Business flights must be timely, cost-efficient, comfortable, 
and enjoyable. Depending on the size of operations, and extent of auxiliary service departments, a 
business aviator’s job can include many other tasks, such as: planning point-to-point transportation and 
accommodations, overseeing cabin cleaning and catering, maintenance, ensuring appropriate facilities at 
destinations for servicing and de-icing, fuel determinations, cost/benefit analysis of fleet augmentation, 
assessing and developing training materials and procedures, and performing passenger briefings and in- 
flight "sight-seeing" tours (Rogers et al. 7998). 

Weather Flying and Business Aviation 

In summary, business aviation operations are a useful platform to consider the effect of weather 
information on piloting for many reasons. 

• Missions can arise on fairly short notice, minimizing the opportunity to carefully watch 
weather trends. 

• Mission destinations vary significantly, reducing the familiarity of pilots with local weather 
phenomena. 

• Mission destinations can be to small airfields that provide little weather advisory information. 

• Mission destinations and flight plans can change dynamically due to passengers' 
requirements. 
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• Passenger comfort and timeliness are extremely important to business aviation. 

• Smaller business aviation aircraft have performance limitations in adverse weather 
conditions. 

1.3 Cognitive Task Analysis 

To affect aviation safety by improving the use of weather information, one must: 1) provide the 
appropriate information to support aviation weather decision-making; 2) present this information in an 
intuitive manner; and 3) aid pilots, especially inexperienced pilots, in gathering information, interpreting 
conditions and selecting appropriate responses. Prior efforts have typically approached this problem by 
asking NAS users to comment on existing, and desired weather information products and services. While 
user suggestions are extremely valuable, and user acceptance is of paramount import, user preferences are 
typically anchored by current conditions and may be inconsistent with performance improvements ( e.g ., 
Antin 1988). Thus, it is imperative that, in addition to valuable user preferences, we more objectively 
ascertain how weather information is used for safe and effective operations in the NAS. In addition to 
ongoing traditional task analyses to objectively identify information requirements, this research used a 
cognitive task analysis (CTA) to better understand the most challenging decisions associated with, and the 
indicators of expertise in, weather flying for business jet pilots. 

Cognitive task analysis differs from traditional task analysis by focusing on operators' cognitive 
processing and knowledge base, or experience. CTAs are typically used when tasks are complex, or ill 
structured; and when these tasks occur in dynamic, uncertain, multi-tasking, real-time operational 
domains (Gordon and Gill 1997). To conduct a CTA, one uses knowledge acquisition tools to elicit and 
represent general and specific knowledge. Typically researchers use these methods with highly 
knowledgeable and experienced operators that are considered subject matter experts (SMEs). The 
knowledge elicitation phase of CTA uses a set of interview techniques to explore these experts' decision- 
making processes. The knowledge representation phase of CTA guides documentation and codification 
of data into formats that support systems design and training recommendations. Many of the CTA 
methods used in this study were adapted from Klein Associates' ACTA (Applied Cognitive Task 
Analysis) toolkit. In addition, this study employed the Critical Decision methodology. These tools and 
techniques are described below as they are generically used. The methods section of this report 
elaborates on how these tools and techniques were adapted for this study. 

Applied Cognitive Task Analysis (ACTA) Methods 

ACTA (Militello & Hutton 1998; Militello, Hutton, Pliske, Knight & Klein 1997) is a streamlined CTA 
method that was developed to be less resource intensive than traditional methods. Traditional CTA 
methods are extremely time- intensive, are therefore typically used with only a few participants, and are 
best used by experienced interviewers. Although ACTA was developed for less-experienced 
practitioners, experienced researchers also use ACTA to obtain a broad understanding of a domain. 

ACTA consists of three interview methods to conduct information elicitation about the cognitive demands 
and skills required for a task. The generic ACTA procedures are briefly described below. Details of the 
ACTA tools are provided in Appendix A. 

The three ACTA interview tools are: the Task Diagram, the Knowledge Audit, and the Simulation 
Interview. The Task Diagram elicits a model of how a SME parses the way they perform a job. The Task 
Diagram results in a set of component tasks for a job, indicates where cognitive skills (interpretation, 
judgements, assessments, problem-solving) predominate, and identifies particularly difficult decision 
points. The Task Diagram structures the remainder of the CTA. The Knowledge Audit focuses on the 
role of expertise in the challenging decisions identified by the Task Diagram. For each of these tasks or 
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decisions, the interviewer asks SMEs to identify how they recognize that they need to make this difficult 
decision, the strategies they employ, and to explain why this is a difficult situation. The Simulation 
Interview provides a view of SME problem solving in the context of actual operation. Context certainly 
guides behavior (Suchman 1987), and as such it is important to provide this context when eliciting SME 
problem-solving processes. Simulation scenarios should be challenging situations that require SMEs to 
exercise their expertise. These scenarios can be provided as text-based descriptions, video depictions, or 
as a scenario in a full-mission operational simulator facility. ACTA developers suggest that high-fidelity 
simulations are not required to obtain useful information. During the Simulation Interview SMEs are 
asked to interpret a situation, explain the cues and strategies they would employ as well as actions they 
would take. Finally SMEs are asked to consider what errors a less experienced and skilled operator might 
make in die simulated situation. 

CTA also requires a format for representing elicited information in order to facilitate using this 
information in design and training. ACTA provides the Cognitive Demand Table as an initial 
representation framework. The Cognitive Demand Table describes why each of the cognitive demands is 
difficult, as well as the cues and strategies used, and errors that may occur in performing it. Other 
representation formats can be developed to support thematic analyses, for example to focus on the role of 
expertise. 

Critical Decision Method 

Klein Associates developed the Critical Decision method based on Flanagan's (1954) critical incident 
technique (Hoffman, Crandall & Shadbolt 1998). The Critical Decision Method uses a specific open- 
ended question to elicit an incident account from a SME. The nature and content of the opening query is 
determined by the research goals of the particular study, but is always asked in terms of an event that the 
SME has personally experienced. For example, in a study of Neonatal Intensive Care Unit nurses' clinical 
judgments, each nurse was asked to select an incident in which her patient assessment skills had made a 
difference to the patient's outcome (Crandall & Getchell-Reiter 1993). In several studies of fire ground 
command decision-making, participants were asked to recall an incident in which their expertise as a fire 
ground commander was particularly challenged (Klein, Calderwood, & Clinton-Cirocco 1988; 
Calderwood, Crandall, & Klein 1987). 

Once a SME identifies a relevant incident, he recounts the episode in its entirety, without interruption. 

The SME's account, solicited in this non- interfering way, focuses and structures the following interview. 
Requesting personal accounts of a specific type of event maximizes response validity, and minimizes 
potential interviewer biases. Once the incident report has been completed, the interviewer revisits the 
incident with the SME several times, using probes designed to focus attention on particular aspects of the 
incident and solicit information about them. Probes elicit details about the recalled event and deepen the 
discussion to provide particular emphasis on perceptual cues ( e.g ., what was actually seen, heard, 
considered, remembered) and strategies employed, rather than general value assessments, explanations or 
rationalizations about performance. Solicited information depends on the purpose of the study, but might 
include presence or absence of salient cues and the nature of those cues, assessment of the situation and 
the basis of that assessment, expectations about how the situation might evolve, goals considered, and 
options evaluated and those chosen. 
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2.0 Methods 


This section characterizes participants in this investigation, provides the CTA protocol adapted from the 
ACTA and Critical Decision methods and explains the approaches used to analyze the resulting interview 
data. 

2.1 Participants 

Business jet pilots were recruited by advertisement in business aviation magazines and by direct contact 
to local businesses. Applicants were screened using the preliminary questionnaire (Appendix B) to find 
participants who had approximately five years of experience as corporate business jet pilots, and who had 
a minimum of 500 hours flying business jet aircraft. After this initial selection, preference was given to 
pilots who fly a wide range of missions, fly at least four or more times each month, and fly to some 
destinations that require three or more hours of flight time. 


Eight male, business jet pilots were selected as participants (Table 2). Pilots’ ages range from 33-61 
years. All pilots are highly experienced and have flown business jets for more than a decade, 
accumulating well over six thousand flying hours in various light, medium, and heavy aircraft. 
Participants flew aircraft which included a variety of Cessna Citation and Gulfstream (II, III, IV, V) jets, 
DeHavilland DHC-8, Rockwell Sabreliner 65; Raytheon Be-200; Learjet 35; Fairchild F27; Dassault 
Falcon, and others. The most predominant type of aircraft flown were light jets (cf. Table 1). 


Table 2. Participant Information 


Participant # 

Age 

Years Business Jet Flying 

Total Hours 

Hours as PIC* 

1 

33 

10 

6800 

2500 

2 

61 

20 

14000 

10000 

3 

37 

15 

6200 

4500 

4 

55 

29 

11500 

6800 

5 

51 

25 

14500 

8600 

6 

50 

7100* 

9700 

3100 

7 

51 

22 

14500 

10000 

8 

40 

14 

11000 

10500 

Median 

50.5 

20 

11250 

7700 

Mean 

47.25 

19.28 

11025 

7000 

* This figure was given in hours and is not included in the summary 

statistic calculations. 


The participants typically fly throughout the United States and North America, and only occasionally fly 
overseas to Europe, Asia, and the Middle East. Participants reported they always have a co-pilot and on 
infrequent, long international flights may carry an extra pilot, flight attendants, and a mechanic. Typical 
flight missions are one-day flights to and from a domestic meeting site. These flights may carry 
customers to corporate offices or may transport corporate executives to customer sites, hi addition to 
passengers, business aviation flights occasionally carry small sales demonstration materials, and aircraft 
parts and equipment. 

Each pilot was interviewed for approximately 3-4 hours and asked to fill out additional questionnaires 
following the interviews 1 . Participants were assured that the content of the interviews would be 
referenced anonymously. All pilots signed a voluntary consent form and were paid $200 plus per diem 
for their participation. 


1 The data from the post-interview questionnaires supports a different study. 
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2.2 Protocol 


In Phase 1 interviews, CTA procedures, materials, and questionnaires were administered to the first three 
participants. The remaining five subjects participated in Phase 2 interviews, using the materials and 
procedures refined based on Phase 1 interview experience and feedback. Interviews were conducted in a 
conference room at the NASA Langley Research Center. Two researchers participated as interviewers for 
each SME interviewed. One researcher led the interview. The other researcher primarily recorded 
participant responses and monitored the first researchers' adherence to the experimental protocol. 
Interviews were audio taped for subsequent transcription and verification of notes. 

Phase 1 Interviews 

An initial set of semi-strucUired interviews was conducted with participants 1, 2, and 3, as SMEs. These 
interviews obtained additional background and mission information from pilots and used the Task 
Diagram, Knowledge Audit, and Critical Decision methods. Appendix C provides the initial interview 
guide. Typically, the Task Diagram method begins with asking participants to note the major tasks in the 
job of interest. For this investigation, these steps were assumed to be the generally accepted phases of 
flight. Participants were provided with a Task Diagram of flight phases (Figure 1), and asked to identify 
the most cognitively challenging tasks that weather information affects. Knowledge Audit probes further 
explored participants' expertise in handling these challenging tasks with challenging weather conditions. 
Participants were then asked to list and characterize the weather information sources they currently use by 
completing the Weather Products Survey (Appendix D). Finally participants were asked, according to the 
Critical Decision method, to recollect a specific weather-related situation in which weather had a role that 
demonstrated their expertise. They were provided the following specific probe: 

“You can probably remember some flights in which you had to make difficult decisions due to the 
current weather conditions. We would like you to describe an incident in which your experience 
made a difference in how you handled the situation — a situation in which a less-experienced pilot 
might have made a different decision than you did.” 

This incident was then further examined to identify their performance goals, decision points, cues and the 
weather information sources they used. 
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Figure 1. Task Diagram: Phases of Flight. 


Phase 2 Interviews 

The second set of interviews was conducted with participants 4, 5, 6, 7, and 8. Phase 1 interview data 
were used to revise the interview guide for the second set of interviews. Appendix E contains the Phase 2 
Interview Guide. Rather than asking participants about their background and mission characteristics in 
the interview, the preliminary questionnaire was augmented to include these queries. The Task Diagram 
and Knowledge Audit methods were employed as previously described, hr addition, pilots were provided 
with a low-fidelity “simulation" to support the Simulation Interview knowledge elicitation technique. 

This simulation was based on actual weather conditions along the Eastern U.S. coast on January 23, 2000. 
Table 3 lists the information provided to participants for this simulation. The constructed materials and 
captured weather products describing weather conditions are provided in Appendix F. A confederate 
researcher, in another conference room, acted as both Flight Service Station (FSS) and Flight Watch (FW) 
personnel. He was provided with scripted responses for anticipated questions as well as scenario weather 
information to answer unanticipated questions. Participants were told to use whatever information they 
had, and to ask for any additional information that they would consult during this mission. Not one 
participant requested information from FSS or FW. While this may have been an artifact of the 
environment, other research shows pilots frequently under-utilize these services in full mission simulation 
experiments as well (Yucknovicz et al. 2000). 
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Table 3. Simulation Interview Materials 


Preflight 

Enroute 

Approach 

Charts 

Scenario Mission 

PEREPS - ride reports 

ATIS recordings 

Approach Charts 

Aircraft Description 

Visual descriptions 

PfREPS - terminal area 

Airport Diagrams 

Standard Briefing 
( DUATs encoded text) 
Weather Graphics: 
Current Surface 
24 Hour Surface 
48 Hour Surface 
Current Flight Rules 
Doppler Radar 
Lifted hidex 
Satellite Image 
Weather Hazards 

FW scripts 

METARS 

IFR enroute low alt. 

IFR high altitude 
Sectional charts 


Essentially, the simulated mission required the pilot to fly passengers to a meeting during questionable 
weather conditions that included icing, limited visibility, and possible high wind conditions. After 
participants read the mission scenario and reviewed the other materials, they were queried for decision 
points. Experimenters further probed these decision points by asking participants to assess the situation at 
the decision point, explain critical cues they would seek, strategies they would use and actions they would 
take. Additionally, participants were asked to describe the difficulty of the decision point, and if difficult, 
how a novice pilot's behavior may differ from that of an expert. 

In addition to the inclusion of a weather simulation, several other changes were made to the interview 
procedure based on the results of the first set of interviews. An additional Knowledge Audit question was 
developed to augment results from the Critical Decision method. This question asked participants to 
conjure a “Scenario from Hell;” that is, a situation that would be extremely challenging, and would 
benefit greatly from expertise and experience. To compensate for the addition of the Simulation 
Interview and "Scenario from Hell" queries, the Weather Products Summary questions were included in 
the Post-Interview Questionnaire (Appendix G). 

2.3 Methods of Analysis and Representation 

Data analyses were based on notes of all interview sessions. While the procedures for Phase 1 and Phase 
2 interviews differed, the authors determined that the effects of these differences were far outweighed by 
the relevance of the resulting data. Information from the interviews was extracted as follows. The scribe 
researcher prepared a detailed summary of the interview content. The interviewing researcher reviewed 
the summary and noted revisions as needed. Audiotapes were consulted and transcribed when researchers 
were unclear about interview notes, or when researchers did not recall the interview content consistently. 
Interview notes and transcripts were then systematically reviewed to develop knowledge representations 
to support analyses. 

Several different data representations were developed. The Cognitive Demands table, as defined by the 
ACTA toolkit, served as the primary knowledge representation, and an adaptation of this table was used 
to represent Simulation Interview results separately. In addition, we developed a model of weather 
information sources and reliability assessment, annotated incidents, and also conducted additional 
thematic analyses from the raw interview data. The methods for developing these representations are 
described below. 
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Cognitive Demand Tables 

Recall that Cognitive Demand tables (CDTs) tabulate the most demanding cognitive aspects of a task, and 
for each of these "demands," identify why it is difficult, the cues and factors that are assessed, strategies 
and actions that would be taken, and what errors novices might make in these situations. We generated a 
CDT for each participant. The CDTs were based on all data obtained across the various knowledge 
elicitation methods. That is, they integrate data from standardized methods (e.g., the Simulation 
Interview) with recollections of personal experiences (the Critical Decision method). Individual CDTs 
represent the cognitive demands identified by each SME according to phase of flight. Sometimes a 
similar cognitive demand was reported in different phases of flight, but different cues or weather 
information were used to make the assessment of the situation, or the same cues may have had a different 
meaning in the different phases of flight. In these cases, the same cognitive demand is listed under all 
relevant flight phases. Appendix H contains these individual CDTs. 

Combining similar cognitive demands across individual CDTs generates two summary CDTs (Appendix 
I). The first table identifies indicated information requirements identified by SMEs for these decisions, 
and summarizes the sources used to obtain weather information and the primary performance goals for 
each decision. The second summary table also lists cognitive demands by phases of flight, but focuses on 
strategies the SMEs employ and errors they suggest novices may make. Together, these tables summarize 
information requirements for flight planning and in-flight decisions and highlight where current 
information sources are inadequate in helping pilots make these decisions. Additionally, they suggest 
challenging weather scenario elements and behavioral markers for erroneous/novice performance in these 
circumstances. 

Simulation Interview Tables 

Simulation Interview data was explicitly represented in a separate table, as well as being incorporated in 
the individual and summary CDTs. Rather than generalized cognitive demands, the Simulation Interview 
tables (Appendix J) presents Phase 2 SME responses to key decision points in a scripted, simulated 
weather-flying scenario. 

Information Sources & Reliability Assessment 

A diagrammatic model of information sources perceived as available to the business jet pilot, and a 
framework for how these pilots assessed information reliability was developed by reviewing individual 
CDTs and responses to the "Your Use of Weather Products" survey. This representation depicts the 
information sources as expressed by the subjects and is not a comprehensive model of all weather 
information that pilots may access. 

Critical Incidents 

Most participants described one or more specific "critical" situations in which their skills as an 
experienced pilot were challenged by weather-related factors. These incidents were intentionally elicited 
using the "Scenario from Hell" inquiry and the Critical Decision method probe. Each incident was 
analyzed to identify the critical decisions contained in the incident and the relevant cues and factors. An 
example of an annotated incident is shown in Table 4. The complete set of annotated Critical Decision 
probed incidents is included in Appendix K. Appendix L describes the "Scenarios from Hell" responses 
from Phase 2 interviews. These analyzed incidents provide actual scenarios that may be recreated in full- 
mission simulation evaluations of aviation information weather systems. 
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Table 4. An Example of an Annotated Incident 


Narrative of Incident 

SME 4 had to miss an approach into a small, unsupported 
airfield near' Traverse City, MI. He could not see the airfield 
even though the ASOS (Automated Surface Observation 
System) was telling him that the airfield was above 
minimums. SME 4 knew that the temperature and dew-point 
spread was close, and drat tire wind was hr tire “wrong 
direction.” But tire automated weather observation equipment 
was telling him that tire airfield was within tire limits. He said 
he “smelled a rat.” He made tire approach, but the visibility 
was only 2 Vi miles and tire ceiling was under 700 feet. He 
could not see tire runway, so he made a missed approach. He 
did not really know how the automated observation equipment 
worked, but he sensed that it might be wrong based on his 
assessment of tire conditions, and his intuition proved him 
correct. He ended up flying to a close alternate airfield. 


Analysis 

Cognitive demand: 

Can I accept low ceiling and limited 
visibility at destination? 

Information sources: 

ASOS at destination; pilot’s observation of 
conditions at destination; pilots’ experience with 
similar weather situations 
Cues: 

temperature/dewpoint spread; wind direction 
Actions: 

attempt approach; observe ceiling and 
visibility 
Cue: 

couldn’t see runway 
Action: 

abort approach; fly to alternate destination 
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3.0 Results 


These results are presented as general observations of expert business jet pilot responses to weather 
flying, and more specifically the cognitive demands and indicators of expertise of these pilots in difficult 
weather flying scenarios. Results were compiled to provide a representation of how these subjects 
perceive the weather information sources available to them, as well as how they assess weather 
information reliability. The results of this study are further interpreted for pragmatic purposes. They 
identify scenario characteristics and behavioral markers to support full mission simulation studies to 
evaluate AWTN systems. They also indicate both design and training interventions to improve business 
jet pilot decision-making for weather flying. 

3.1 General Observations 


Based on the entire corpus of data, we observed these general findings: 

• Twenty-two weather-related critical decisions were identified for the business jet pilot 
community. Weather conditions impact decisions at all phases of flight and levels of 
decision-making, from general planning to tactical runway selection in changing winds. 

• Three high-level pilot goals were identified: Flight Safety, Flight Efficiency, and Passenger 
Satisfaction. The pilots we interviewed emphasized that flight safety is never compromised 
to improve efficiency or passenger satisfaction. However, these pilots note that these goals 
are all influenced by weather and can potentially conflict, potentially requiring in-flight re- 
planning and communication with passengers. 

• Individual differences were observed across the different pilots we interviewed. They varied 
in the amount of effort and time they expended to understand the weather picture in the 
Simulation Interview, and in their weather information acquisition and en route decision- 
making. There were also differences in the degree to which they relied on reported weather 
information (such as METARs, TAFs, PIREPs, ATC, ATIS, etc.) versus their own senses. 

• Business jet pilots seek out and rely on the most timely weather information. The older the 
data, information, or forecast, or the further away from the pilot's position, the less reliable it 
is perceived to be. Business jet pilots tend to rely on the reports of experienced humans over 
automated or processed data and information. They have the most confidence in what they 
can observe directly. 

• Weather-related decisions made by business jet pilots differ from those made by general 
aviation and transport pilots due to differences in mission and aircraft characteristics. 
Business jet pilots have more flexibility in achieving their mission than do transport pilots. 
They can accommodate weather situations by leaving earlier/later, flying into a different 
airfield, detouring around a wider area, etc. However, they have impoverished weather 
information to the extent that fewer have dedicated weather ground services, and they may 
have to fly into unsupported airfields. Business jet pilots tend to rely more on their piloting 
skills, and on the higher performance capabilities of their aircraft and possibly auxiliary 
equipment (e.g., onboard weather radar) to tactically maneuver around weather they 
encounter. The pilots we interviewed suggested that they would be less likely to do this, and 
be more reliant on preflight and strategic weather information, when flying smaller general 
aviation aircraft. 
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• In the Simulation Interview, most SMEs exhibited frustration with the standard FAA/ICAO 
encoded DU ATS text preflight information. Many admitted they did not typically use this 
type of preflight information. Many of our participants were accustomed to having tailored 
weather briefs from private weather providers, and were not facile with standard DU ATS text 
reports. Reports tailored to the departure point, route of flight, and destination supported by 
graphical representations were preferred. 

• Experienced business jet pilots are able to accurately judge the severity and dynamic nature 
of the weather most of the time. They have a good understanding of the performance 
characteristics of their aircraft and are typically able to “fly around” significant weather 
situations. They have well- developed self-monitoring skills that enable them to make 
effective judgments about their ability to fly in difficult weather conditions. 

3.2 Cognitive Demands 

The identified cognitive demands and decisions faced by business jet pilots in making weather-related 
decisions and judgments are described below according to these flight phases: preflight planning, taxi & 
take-off, climb, cruise, descent & approach, land & taxi. 

Preflight Planning 

• What’s the general weather picture? Will it be affecting my flying in the next few days? 

This judgment is a very broad, early, assessment of expected weather problems for operations in 
the subsequent day or days, and the kinds of weather phenomena that the pilot will need to pay 
attention to for detailed planning. This judgment relies heavily on knowledge of atmospheric 
dynamics and regional trends. 


• What are the weather problems for this flight? 

Departure: Is it a Go/No Go ? VFR/IFR? Will the weather affect my departure time? 

Pilots consider whether the weather is severe enough to cancel the mission. Concerns about 
weather hazards for a mission lead pilots to consider alternate flight plans. For business 
aviation operations, this requires a conference with passengers to determine viable options for 
alternative departure times and/or destinations. In special circumstances, this requires 
planning for different departure points. For example, one study participant flew under Visual 
Flight Rules (VFR) conditions from the location of his business jet’s hanger to a small private 
airfield in order to pick up the passengers. If he anticipated weather conditions would not 
support the VFR portion of his mission, he would have to arrange to meet his passengers at 
the public airfield. Weather also affects the flight rules that a mission is operated under. 
Considering the weather in context of departure concerns leads pilots to consider general 
problems that can occur in taxiing, takeoff, and climb. These considerations include, for 
example, the effect of possible icing on climb performance. 

En route: Which route will I take? Will I need to detour around weather? 

This includes decisions relating to the planned route of flight, potential detours and added 
time required to reach the destination, any specific safety concerns, etc. 

Destination: Is it still viable? Do I need to consider alternative destinations? If so, where? 
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This includes planning decisions related to the destination itself. These include: Is it even 
feasible to land at the preferred airfield? Should I consider an alternative now and/or plan to 
deviate en route? If so, which will be the most feasible and convenient for my passengers? 


• How does the weather affect the fuel I carry? 

Pilots assess whether extra fuel will be needed for possible deviations from route, delays, holds, 
or diversions to an alternate airport due to weather. The desire to carry additional fuel for mission 
flexibility must be considered in light of the costs of doing so. These costs include: expense of 
extra fuel, and reduced aircraft climb-rate and payload capability. 

• How much confidence do I have in the co-pilot's weather flying abilities? 

This assessment affects how comfortable a pilot might be in making a mission, the workload 
distribution he expects during the mission, and how often he might engage in monitoring and 
cross-checking his co-pilot. This assessment is based on familiarity with the co-pilot, knowledge 
of their general flying experience as well as region-specific and weather-exposure experience. 

Taxi & Take-off 

• Can I taxi/take off in this weather? 

The weather concerns relate directly to the ability of the pilot to safely taxi and takeoff, but also 
affect passenger comfort and schedule. Pilots consider weather factors such as ceiling, visibility, 
crosswind, windshear, runway conditions, and icing probability, in these assessments. 

• Will I need to consider a rejected takeoff? 

Pilots consider the likelihood of having to abort a takeoff and the contingency actions if a takeoff 
is aborted. Cross wind speed and direction, thunderstorms, runway conditions, and other weather 
phenomena can influence these plans. 

• Do I need to delay my takeoff? 

Sometimes weather conditions can change significantly between preflight planning and actual 
departure. Pilots must reconsider the safety of departing at the planned time, and may consider 
delaying departure. As previously mentioned, such a decision requires a conference with 
passengers to preserve, if possible, success of the ultimate business mission. Delays can also be 
required to ensure that the aircraft is properly configured for weather conditions. Deicing, and 
ground use of the onboard weather radar are specific examples of such delays. 

• Do I need to plan post-takeoff actions? 

Pilots think beyond takeoff to plan for actions based on weather factors that may impact the 
aircraft immediately during the takeoff. For example, they may plan a specific maneuver to avoid 
a weather cell in the vicinity. They may consider wind conditions when planning for engine 
failure contingencies. 

Climb 

• Do I anticipate icing problems? 

Pilots anticipate the likelihood of icing on the climb-out and judge the impact of icing on flight 
safety and aircraft performance. The pilot must continually be aware of the aircraft’s 
performance in potential icing conditions. 

• Do I anticipate wind-related problems (turbulence, windshear, crosswinds)? 
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Pilots are also aware of wind conditions that impact aircraft performance and ability to maintain 
lift. Pilots also consider wind conditions’ effects on passenger comfort. 


• Do I anticipate any weather obstacles on climb out? 

Hazardous weather phenomena, such as convective cells, rain shafts, wind shear activity, and 
reported lightning create obstacles that the pilot tries to avoid, for safety and passenger comfort 
reasons. Deviations around these obstacles should be considered in advance and with respect to 
other constraints such as terrain and airspace restrictions. 


Cruise 


• Is my destination still viable? 

These decisions relate directly to judgments and assessments made during the cruise phase of 
flight. If the weather at the destination is uncertain or less than anticipated, pilots will more 
frequently sample weather data for the destination during cruise. This information is used to 
continuously assess the viability of the destination, and alternate destinations. Viable alternates 
must be appropriate not only in terms of weather conditions, but also for the aircraft's fuel 
available, and performance capabilities. Further, alternates should be acceptable for the business 
mission (i.e., passengers will still be able to accomplish the business purpose of the mission). 

• Is the flight proceeding according to expectations? 

This is a general assessment of actual progress against planned progress; of unexpected 
occurrences or weather, of planned ground speed versus actual ground speed (unexpected 
headwinds aloft), of estimated-time-of-arrival (ETA) versus the passengers’ schedules. The 
pilots we interviewed mentioned using PIREPs as well as checking with FSS and ATIS just after 
the hour, when new information may be available. Deviations from expected occurrences and 
performance spur pilots to acquire additional information about environmental conditions and 
engage in replanning. Pilots may also decide they need to alert their passengers if the flight is not 
proceeding as planned (e.g., ETA changes). 

• If I see an unexpected weather hazard, how can I avoid it? 

Perfect weather information would enable perfect strategic flight planning to avoid weather 
hazards. However, because weather dynamics cannot be perfectly modeled and therefore 
forecasted, and because weather information may not be immediate or precise enough, some 
pilots "wait and see" what the weather looks like before attempting to avoid it. Tactical 
avoidance strategies include flying around or over the weather, and most conservatively, simply 
turning around. Tactical avoidance of weather hazards requires pilots to consider weather 
dynamics and intensity, aircraft performance characteristics, and other airspace constraints (e.g., 
terrain, airspace). In addition to simply tactically avoiding weather they could see, several 
interviews suggested that pilots query ATC to better understand how pilots closer to the 
anticipated hazard are choosing to avoid it. 

• How can I provide passengers with the most comfortable ride? 

En route decisions also must consider the impact on passengers: Are they comfortable? Can they 
work or sleep comfortably? Turbulence from weather cells, and the jet stream is the primary 
concern with regard to passenger comfort. If turbulence is unavoidable, pilots attempt to notify 
passengers prior to onset. Advance notice of such turbulence is therefore valuable information. 
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Descent & Approach 


• Do I anticipate significant weather on descent? 

When in the vicinity of the destination airfield, pilots obtain local weather information (e.g., 
ceiling, visibility) and reconsider the safety of landing at this airfield and on the designated 
runway. Pilots consider suboptimal conditions (e.g., icing, windshear, turbulence, crosswinds, 
runway conditions) that require additional contingency planning for missed-approach planning. 

• How do I deal with weather hazards on descent? 

This judgment is difficult because the options narrow for the pilot as the airfield gets closer. 
Different obstacles require different avoidance strategies. Also, the closer to the ground the 
aircraft is, the more severe the consequences of loss of lift. Expert pilots develop these strategies 
in advance and consider airspace and terrain, as well as aircraft performance characteristics. 

• Do I want to hold or divert to an alternate destination? 

This decision may be instigated by weather conditions, but must also be evaluated in light of 
weather conditions. While one might decide to hold to see if an airfield's visibility improves, it is 
unwise to hold in what may be icing conditions. The decision to hold or divert also affects 
passengers’ schedule and ground transportation logistics; and requires additional fuel and 
therefore expense. If the pilot chooses to not carry extra fuel, for improved payload capacity or 
climb performance, s/he may eliminate the option of holding at the destination. Provided safety 
is not compromised, decisions regarding changes to the final destination are often shared with 
passengers. 

• Do I anticipate a missed approach and go-around? 

Pilots consider the likelihood of a missed approach or go-around due to weather conditions, and 
the effect of weather conditions on the ability to perform these maneuvers. 

• What kind of approach do I need to make (Visual, ILS)? Which runway? Is it suitable? 

This more specific judgment addresses selection of approach and runway. Even if local weather 
information defines the airfield conditions to be safe, pilots still must consider the 
appropriateness of the approach and runway for the weather conditions. For example, ice- 
contaminated runways and runway length must be considered in light of aircraft landing 
requirements. 


Landing & Taxiing 

• What kind of landing do I need to make? 

This decision is made based primarily on safety factors, although it influences passenger comfort. 
If the runway is very wet, it may be necessary to bring the aircraft down hard to avoid skidding 
on the wet surface. To inform such decisions, the pilot needs to have an accurate appraisal of the 
runway surface conditions and surface winds. 

• Do I need to prepare the aircraft for a quick turn-around for my next leg/trip? 

This cognitive requirement is faced when the aircraft has to be turned around quickly for the next 
leg of a trip. Pilots must think about how the weather will change while they are on the ground 
and how they may have to accommodate that into their plans. 
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3.3 Indicators of Expertise 

Consideration of expert/novice performance differences is useful to design supportive aiding technologies 
and training programs. In addition, these differences and the errors that novices may make define 
behavioral markers for assessing performance. In this study, we only interviewed experienced business 
jet pilots, so our conclusions about these differences are derived from these individuals' perceptions of 
their own skill development and contrasts of their behavior with less experienced colleagues. Study 
participants were specifically asked to consider the types of errors less-experienced pilots might commit. 
The benefits of piloting expertise and deficiencies of novices are as follows: 

• Expert business jet pilots accurately judge the severity and dynamic nature of the weather most of 
the time. Novice business jet pilots are more likely to misinterpret weather severity or fail to 
anticipate changing weather patterns. Novices, therefore, may not be able to adjust properly to 
changing conditions, either disrupting safety, efficiency, and/or passenger comfort. 

• Expert business jet pilots build a dynamic, "big picture" of a situation by understanding how a 
situation developed, assessing relevant current and forecasted data, and projecting the situation. 
Novice business jet pilots build incorrect or incomplete pictures by either not thinking far enough 
into the fuUtre (e.g., not contingency-planning), failing to continuously monitor changing weather 
conditions, and failing to use all relevant resources to check the reliability among them. 

• Expert business jet pilots have a good understanding of the performance of their aircraft and 
onboard equipment, the limitations imposed by weather conditions, and strategies for recognizing 
and compensating for these limitations. Novice business jet pilots may not consider how changing 
weather affects aircraft performance and therefore are forced into a reactive mode of control. 
Novices may not have a refined knowledge of how to use instruments to their fullest potential to 
best acquire weather information and recognize effects of weather on aircraft performance. 

• Expert business jet pilots have strong self-monitoring skills that enable them to judge when they 
should avoid weather rather than manage through it. They more accurately judge their own 
abilities to fly in certain conditions, and more readily catch their own judgment errors before they 
are enacted. Novice pilots who have not experienced difficult weather flying, have not developed 
diagnostic skills for identifying hazards or decision-making skills for selecting appropriate 
responses, can have a cavalier attitude and venture into hazardous situations. This deficiency in 
skills, in conjunction with the delayed realization that one is in a novel and hazardous situation, 
can quickly overwhelm and confuse a novice pilot. 

• Expert business jet pilots are more aware and attentive to all details of flight, from passenger 
schedules to atmospheric pressure changes to providing passengers services such as catering. 
Novices tend to be more focused on piloting and making the next point in the flight plan. 


3.4 Weather Information Sources Diagram 

The Weather Information Sources diagram (Figure 2) represents all the sources of weather information 
and types of weather information that the SMEs referred to in interviews. The diagram does not represent 
all possible sources of weather information that are available to the pilot community. The sources 
depicted are those that are most commonly reported by the business jet pilots we interviewed to help them 
understand the weather picture and its implications for flying. These sources, the ovals, are described in 
terms of the weather information, the rectangles, for which they are referenced. 
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3.5 Assessing Weather Information Reliability 

The Weather Information Sources diagram illustrates that pilots use a variety of weather information 
sources to arrive at their "weather picture." SME responses also were analyzed to determine the 
characteristics that these pilots use to assess the reliability of weather information, and thereby prioritize 
weather source usage. These interrelated factors include: directness of perception; timeliness and useful- 
life of data; spatial dislocation of the pilot/sensor from the weather, sensor and trans mi ssion equipment 
quality, and spatial resolution, and meta-cognitive interpretive skills. 

Pilots in this study indicated that they are more likely to trust their own direct perceptions than those of 
others, or of automatically-sensed information, and more likely to trust the perceptions of other humans in 
the NAS than that of automated weather information services or sensor-driven displays. Of course not all 
weather information of significance is available to direct visual perception. To the degree that presented 
data is isomorphic with the atmospheric conditions, it is more "direct." For example, a satellite picture of 
cloud cover is considered more direct than a mosaic radar return of the same area. 

The timeliness and useful-life of the data are interrelated. Subjects assessed the credibility and reliability 
of current weather reports based on how long ago the reading was taken, and on the dynamism of the 
observed weather phenomenon. For example, a report of a low-lying fog over an airfield may be valid for 
several hours after the report was made. Conversely, a turbulence report at flight level 310, may rapidly 
become obsolete or may be valid for several hours, depending on the weather phenomenon's dynamics. 
Generally speaking, pilots trust weather reports that are more recent than those that are less recent, and 
use their knowledge of weather dynamics to assess when reports are likely to be obsolete. Weather 
forecasts are similarly assessed. Pilots are unlikely to rely on forecasts to the degree that the weather they 
predict is dynamic. Some subjects in this study mentioned that they assess the validity of forecast 
information by comparing past forecasts and current conditions. 

SMEs in this study also mentioned spatial dislocation of the sensor (either human or automated) from the 
phenomenon as a significant factor affecting their assessment of weather information validity. In essence, 
this factor can be described as either due to the effects of the prior factor, "timeliness and shelf-life" vis a 
vis atmospheric dynamics, or to inaccuracies that increase in sensor and transmission capabilities with 
increased distance. This factor is described next. 

SMEs are aware that the weather information available to them is only as good as the equipment used to 
detect, transmit, and present this information. Directly perceived weather information is subject to, for 
example, the limitations of visual perception. Similarly, for automated sensing systems, the sensor 
quality (sensitivity, false alarm rate) and the transmission quality (timeliness and completeness of 
uplinks) must be considered when assessing the validity of the resulting weather information. Both 
human vision and automated sensors are likely to be degraded with distance. Many of the pilots 
interviewed reported that their onboard radar provided varying degrees of reliable and accurate data. 

These data are dependent on the range of the phenomenon from the aircraft, the power of the radar, the 
radar range setting, the presence of atmospheric or ground clutter, and the intensity of the weather 
phenomenon itself (which may block the extent of the radar's effective range). 

The spatial resolution of the weather information is relevant to SMEs reliance on this information. These 
pilots rely on weather information to the degree that it is spatially precise and appropriate for their 
flightplan. For example, the general outlook for a 24-hour forecast may predict “half-mile visibility in the 
vicinity of Greenville Airport” but a pilot may still fly out because the hazard is localized to the South 
side of the airport. This forecast does not provide the relevant level of detail to fully trust this information 
to make such a decision. 
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Finally, meta-cognitive skills are important in determining how pilots use weather information sources. 
The subjects in this study demonstrated a range of comfort in understanding atmospheric phenomena, 
weather hazards and dynamics, and technical details of weather information sensors and services. These 
pilots determined appropriate reliance on abstracted and sensed weather information sources to the degree 
that they are confident in their weather assessment skills. Their self-confidence in interpreting weather 
data, detecting false indications in weather products and predicting weather behavior allowed them to 
more intelligently assess the reliability of individual reports and forecasts. These assessments were often 
derived from regional experience and history of exposure to the interaction of atmospheric conditions and 
terrain. Some of our participants made their own assessments of raw information, and generated their 
own forecasts. They were able to generate a reasonable explanation of the weather in certain areas for 
certain periods. If the existing weather data, information, and forecasts matched their own predictions, 
they could place more faith in it. They “owned” their weather data, rather than relying on someone 
telling them what was happening and what was going to happen. Less meteorologically skilled 
participants were more likely to either blindly trust preflight weather information or tended to completely 
dismiss this information and be more reliant on, and reactive to, in situ direct and PIREP observations. 


3.6 Scenarios and Measures for AWIN Simulation Experiments 

AWIN researchers will be evaluating aviation weather information systems in flight experiments and full- 
mission simulation evaluations. One critical facet of such experiments will be defining, and in the case of 
simulation evaluations designing, scenarios that stress pilots' weather-flying skills and decision-making. 

It is equally important to evaluate pilot performance using sensitive, operationally significant measures. 
Data obtained from the Critical Decision and "Scenario from Hell" probes provide a foundation for 
defining scenario elements that stress weather-flying skills and decision-making and indications of good 
performance in these conditions. Cognitive Demand tables provide behavioral markers of novice pilot 
errors. 

Appendix M parses Critical Decision and "Scenario from Hell" incidents into scenario elements. These 
elements are further classified by phase of flight, type of element (weather condition, system malfunction, 
environment, and infrastructure), the sources of information to cue the pilot, and behavioral markers for 
good performance. Some weather situations that may be used as scenario elements include: 

• an uncontrolled airport as a destination with weather near minimums and non-precision 
approaches, 

• unreliable or uncertified AWOS, 

• highly dynamic weather conditions at the destination, 

• difficult crosswinds on the assigned runway, 

• weather cell in vicinity of airport and further constraints from terrain, 

• ground sensors or personnel provide optimistic weather intensity and closure rate 
assessments, 

• runway ice contamination in conjunction with anti-skid failure, 

• engine failure at take-off with cross winds, 

• bleed air failure and icing conditions, 

• icing during a night flight with pitot-static occlusion, 

• onboard radar clutter and attenuation, 

• dense fog occurs just after reaching rotate speed. 

Subjects mentioned other factors that are relevant to business jet operations that exacerbate bad weather 
condition situations. These include: 

• obvious alternates that don't provide sufficient ground transportation for passengers, 
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• late and anxious passengers, 

• uncooperative co-pilot, 

• ground switches runways at last minute, 

The expert subjects in this study were specifically asked to consider what errors novices would make in 
challenging weather flying situations. These data are presented in their entirety in the Critical Decision 
Tables. A summary of these novice aviator errors follows: 

• inappropriate fuel loading: failure to consider need to reach alternates & performance 
implications 

• over-reliance on forecasted weather in dynamic conditions 

• failure to intermittently monitor changing weather conditions at appropriate sampling rate, 

• failure to acquire information from multiple weather sources to cross check for validity, 

• over-reliance on automatically sensed weather information, 

• acquiescing to passenger pressure to make trip when concerned about weather, 

• failure/inability to develop trend information; to predict if conditions are worsening or 
improving, 

• failure/inability to consider, temporally project weather intensity and location, 

• failure to appreciate limitations of, and/or appropriately manipulate onboard radar sensors, 

• insufficient understanding of aircraft performance and implications for weather avoidance, 

• failure to appreciate effects of weather, and weather equipment use, on aircraft performance, 

• failure to accurately predict effects of weather on passenger safety and comfort, 

• failure to continue to monitor fuel in weather avoidance, 

• staying on a visual flight plan (VFR) in instrument meteorological conditions (IMC), 

• over-confidence in abilities, 

• failure to assess runway crosswinds until too close in. 


3.7 Design & Training Interventions 

The cognitive task analysis of business jet pilot weather-related decision-making generated several 
recommendations for this niche in the flying community. Some of the recommendations are very specific 
to the business jet community, whereas others are probably applicable across other groups within the 
flying community. We have identified two areas for our recommendations: business jet flight deck 
design and business jet pilot training. 

Design Interventions 

The following recommendations address system design issues for the business jet pilot community. They 
address only issues specific to decisions and judgments related to weather. 

■ Integrated Source Displays. Currently, pilots must integrate multiple types of information from 
multiple sources. These include up-linked radar pictures, up-linked satellite, textual ATIS 
reports, voice reports of turbulence over common broadcast radio frequencies, automated 
observations from specific locations, weather alerts issued by various agencies, etc. Flight deck 
displays should provide an integrated representation of weather information to support the 
cognitive demands of situation assessment and response planning. 

■ Big Picture Radar Representations. Due to the flexibility required of business jet missions, it 
is particularly important that these pilots understand the big picture of the weather situation. 
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Augmenting onboard aircraft radar capabilities with up-linked "big picture" radar information 
would aid challenging en route re-routing decisions. 

■ Radar Image and Usage. Pilots need weather hazard displays, rather than radar return displays. 
Subject matter experts should be used to develop rules for interpreting onboard weather radar data 
and rules for understanding when that data might be suspect. In addition, subject matter experts 
could serve as the basis for codified guidance on how to more expertly and sensitively manipulate 
the onboard weather radar system to gather the most appropriate weather information, as well as 
be able to assess the reliability of that information. Onboard weather radar systems, as well as on 
board aviation weather information systems more generally, would benefit pilots by relieving 
workload if they performed automated monitoring for weather hazards and changes, and provided 
indications of these events in a manner consistent with their import. 

■ Significant Weather Alerts. Participants frequently mentioned problems associated with 
unreliable weather information in terms of timeliness, and relevance in terms of proximity to their 
own position. Pilots do not always know when to seek information, or when the weather 
situation changes, for better or worse. Remembering to obtain weather information and actively 
doing so are significant workload impositions on a pilot. Onboard aviation weather information 
systems as well as ground-based weather information systems, would benefit pilots by relieving 
workload if they performed automated monitoring for relevant weather hazards and changes, and 
provided indications of these events in a manner consistent with their import. Currently, ATC 
broadcasts weather alerts to pilots on IFR flight plans or who are using VFR flight following. 
However, these alerts are weather-centered, i.e., they describe the location and severity of 
weather, not pilot-centered, i.e., considering a specific route, aircraft performance capabilities, 
etc. These weather-centered reports provide necessary information, but in a manner that has the 
following disadvantages: they may be irrelevant to a particular route and/or aircraft, they may be 
delayed, and they rely on radio frequency availability - which becomes saturated in difficult 
weather conditions. 

■ Decision-Centered, Action-Oriented Alerts. The goal of in-flight aiding systems should not 
necessarily be to tell pilots what to do, but to support their own situation assessment and decision- 
making process. One aspect of this support is to provide interpretations and implications of 
different weather situations on the pilot’s intended course of action. For example, by considering 
the implications of weather alerts to the possible alternative runways in an approach and landing 
decision, one might receive a warning such as: 

“Crosswinds at destination X conflict with chosen runway Y. . 

Wind speed and direction are above aircraft landing minimums. . . 

Selected runway Y is now marginal or non-viable. 

Suggest runway X, or suggest alternate airfield .” 

This warning provides the current wind conditions at the destination airfield in relation to the 
intended landing runway, indicates the current status of the runway for landing, and suggests 
alternate courses of action. Ultimately, pilots would be responsible for acting on this information, 
or they may choose to wait, and reassess the intended runway conditions when closer to the 
airport. 

■ Winds Aloft Information System. Pilots reported that winds aloft information is particularly 
unreliable. This information is critical to business jet pilots who are trying to maximize 
passenger satisfaction and flight efficiency. Pilots suggested that tire winds aloft forecasts could 
be improved by using data from aircraft in the air that are sensing wind speed and direction in 
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real-time. These data could be transmitted to a central location, aggregated and integrated into 
current wind readings. This would improve current observations of winds aloft that are currently 
collected by the 12-hour weather balloon recordings. 

■ Weather-Integrated Route Planning Decision Support. One of the challenges faced by the 
business jet community is to understand the impact of weather on their route of choice and to 
select alternate routes based on this information. Because the business jet community has flexible 
routing and scheduling, route planning is more challenging than it is in the transport aviation 
world. One solution would be to integrate route selection planning with current weather 
information to provide the most feasible routes given current weather hazards. Pilots reported 
that the flight planning systems they typically use calculate routes to their destination without 
taking into account the current weather situation. Only after the routes are planned are weather 
concerns superimposed on this route and the route evaluated for safety. An improved flight 
planning system would incorporate routing around such weather hazards and consider the 
dynamics of weather system movement and intensity, and consider these hazards in conjunction 
with terrain and airport capacity constraints. 

■ Tailored Weather Planning Information. Many private weather information providers provide 
the aviation community with weather information tailored to specific routes and aircraft 
capabilities. These tailored weather briefings are preferred over standard DU ATS text preflight 
information. Pilots expressed preference for graphical weather information. Reference locations 
used in AIRMETS and SIGMETS are often difficult to interpret in terms of the pilot's route, a 
problem exacerbated when a pilot is unfamiliar with the region. 

■ Anomaly Detection Support. Pilots would benefit from decision support that would flag 
deviations from expected flight progress or weather status. Rather than a wide area broadcast, 
this information would be specific to a mission. It would relate to the current flight parameters 
( e.g ., route and ETA), and would reflect any changes above a “threshold” (to be empirically 
determined or user-set), to alert pilots that their situation awareness is based on old information, 
and therefore would more sensitively alert them to slowly developing problems that may 
otherwise go undetected. For example, an alert would draw the attention of the pilot to variations 
from the ETA based on projected route, speed, altitude, and wind conditions. If any of these 
parameters altered the ETA significantly the pilot could be alerted and the winds aloft projections 
could be recalculated in order to assess their fuel situation. Aiding could monitor and assess the 
impact of de-icing equipment on aircraft performance and alert the pilot to any significant 
impacts on fuel consumption. 


Training Interventions 

The following recommendations address training issues for the business jet pilot community. They 
address only issues specific to decision-making and judgment related to weather. 

■ Radar Usage. Onboard weather radar provides very helpful, real-time information, however, 
pilots are not adequately or routinely trained to manage this technology or interpret radar return 
images. Previously, we recommended aiding technologies to support these management and 
interpretation tasks. This recommendation does not obviate the necessity of training pilots to 
manually manage radar, interpret radar images, and, in particular, understand the limitations of 
this technology. 
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■ Training Scenarios. During training, weather problems are often provided as a single obstacle 
that introduces sudden flight problems. Training scenarios should also address more complex, 
often seemingly benign, weather situations that may be encountered by pilots. These scenarios 
should include conflicting passenger comfort and flight efficiency goals based on the 
interpretation and assessment of uncertain weather conditions. Scenarios should also present 
situations that have combinations of weather-related problems, and combinations of weather and 
equipment failure scenario elements. Training scenarios should be designed to sharpen the pilot’s 
ability to gather weather information, interpret the weather situation, identify the time course of 
weather dynamics and response requirements, and select and evaluate options. 

■ Goal Deconfliction. Training for the business jet pilot community needs to focus on the 
efficiency and comfort goals as well as the safety goals. The pilots we interviewed did not 
describe situations in which they had compromised the safety of their passengers. Many of the 
difficult judgments pilots reported were related to conflicts between organizational and 
passenger-related goals. For example a participant reported “Ceiling and visibility restrictions 
may not allow me to get my clients to their preferred destination — should I go ahead and attempt 
the trip, even though I may have to turn around and bring them back to the departure point?” 
Pilots could be explicitly trained to identify goal conflicts, and effectively use weather 
information to evaluate options. Scenarios designed to pit these goals against each other or 
introduce factors that may generate several conflicts within a goal would help the pilots make 
better use of their weather products, recognize situations where these conflicts are likely to arise, 
and develop strategies for identifying the best course of action. 
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4.0 Conclusions 


This study conducted a cognitive task analysis with eight expert business jet aviators to better understand 
challenging weather-related decisions. Several CTA methods, developed by Klein Associates were used. 
In addition, we extended the Simulation Interview method by providing participants with a description, 
and weather information materials from an actual challenging weather day. 

Several representations were developed to codify interview data and facilitate analyses. Results distilled 
to 22 different cognitive demand categories faced by experienced business jet pilots related to weather 
conditions. The expert participants suggested how novice pilots might have difficulty with these 
challenging conditions. We developed a representation for the information sources the participants in this 
study mentioned while describing challenging weather-flying situations, and developed a framework to 
describe the factors that these pilots consider when evaluating trust in weather information sources. 
Results were considered in light of supporting empirical simulation evaluations, through scenario and 
measurement development, and in light of identifying design and training interventions. 

The objective of this report is to document the CTA methodology developed for this study and to present 
these preliminary findings. While many CTAs are conducted with as few as eight SMEs, the complexity 
of the flight mission and variety of themes addressed in this study necessitated a broader treatment in the 
interview. For this reason, additional CTAs should be conducted with the documented protocol to fortify, 
or perhaps extend, these preliminary findings. 
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5.0 Acronyms & Abbreviations 


ACARS - Aircraft Communications and Reporting System. 

ACTA - Applied Cognitive Task Analysis toolkit (copyright, Klein Associates, Inc.). 
AIM - Aeronautical Information Manual. 

AIRMET- Airman’s Meteorological Information. 

ASIST - Aeronautics Safety Investment Strategy Team. 

ASOS - Automated Surface Observing System. 

ATC - Air Traffic Control. 

ATIS - Automatic Terminal Information Service. 

AWIN - Aviation Weather Information. 

AWOS - Automated Weather Observing System. 

CDT - Cognitive Demands Table. 

CTA - Cognitive Task Analysis. 

DUATS - Direct User Access Terminal System. 

ETA - Estimated Time of Arrival. 

FBO - Fixed Base Operator. 

FMS - Flight Management System 
FSS - Flight Service Station. 

FW - Flight Watch. 

GPWS - Ground Proximity Warning System 
IFR - Instrument Flight Rules. 

IMC - Instrument Meteorological Conditions. 

METARs - roughly translates from French as Aviation Routine Weather Report. 
NBAA - National Business Aviation Association. 

PIREP- Pilot Report. 

SIGMET - Significant Meteorological Information. 

SME - Subject Matter Expert. 

SPECI - roughly translates as Aviation Selected Special Weather Report. 

TAF - Terminal Area Forecast. 

NAS - National Aerospace System. 

NASA - National Aeronautics and Space Administration. 

TV - Television. 

UNICOM - Universal Communications station. 

VFR - Visual Flight Rules. 

WSI - Weather Services International, Corp. 

WX- Weather. 
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7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.) 


ACTA - TASK DIAGRAM 

Purpose: The Task Diagram is intended to serve as a road map to the rest of the CTA. The Task 
Diagram acts as an advance organizer, providing an overview of the task and identifying the 
cognitively complex elements of the task. 

How to get started: Before you begin, have clearly hi mind what the task is you intend to investigate. 
In this interview, you want to find out about the interviewee’s processes as they perform the task of 
interest. 

CONDUCTING THE TASK DIAGRAM INTERVIEW 

* Write the Task of Interest at top of whiteboard. 

* Elicit the steps required to do the task. Record them across the board from left to right in 
chronological order. Use arrows to indicate the order in which the steps occur. 

- Ask your SME, “Think about what you do when you (Task of Interest) . 

Can you break this task down into between three and six steps?” 

* Elicit information regarding which of the steps require cognitive skills. Circle the elements that 
require cognitive skills. 

- Ask your SME, “Of the steps you have just identified which require difficult cognitive skills? By 
cognitive skills I mean judgments, assessments, problem solving — thinking skills.” 

At this point, you should have a very broad overview of the task, with an indication of where the 
complex cognitive skills lie. If the task seems too big or the steps you have identified are too broad for 
further mvestigation, you may choose to focus on one or two of the subtasks you have identified as 
requiring cognitive skills. In this case, you should complete a Task Diagram on the step(s) you have 
chosen to focus the rest of the cognitive task analysis. 


TIPS FOR DOING THE TASK DIAGRAM INTERVIEW 

* Your interviewee may immediately start talking at a very fine level of detail. Make it clear early on 
that you are looking for a very broad overview with this interview. You will be interested in 
hearing lots of stories and details later in the session (with the Knowledge Audit and the 
Simulation Interview). 

* If your interviewee begins listing things to consider rather than the steps of the task, help reframe 
the topic for him/her. “ What do you do when you (Task of Interest) ?” 

* This may be a new way for the interviewee to think about the job. Give him/her tune to think. You 
may need to repeat or rephrase the question. 

* The Task Diagram serves as a road map to the rest of the cognitive task analysis. You are not trying 
to elicit detailed, specific cognitive information with this interview. You are trying to get a sense of 
which parts of the task require complex cognitive skills. 
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7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.] 

EXAMPLE: Task Diagram of Fireground Commander's Job in Commanding Crew 



The interviewer decides this is too broad — really wants to focus on the assignment of tasks 
during an incident. 


EXAMPLE: Task Diagram of Assign Tasks 
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7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.) 


ACTA - KNOWLEDGE AUDIT 

Purpose: The Knowledge Audit provides details and examples of cognitive elements of expertise; it 
contrasts what experts know and novices don’t. 

How to get started: You used the Task Diagram to identify pails of the job that require skilled 
judgment, decision-making and evaluation. In the 
Knowledge Audit you will elicit the expertise necessary to 
do each of those tasks. Use the Task Diagram to help you 
decide which tasks and subtasks you want to explore with 
the Knowledge Audit. Go into the Knowledge Audit 
interview knowing what you want to analyze. 

CONDUCTING THE KNOWLEDGE AUDIT 

* Write the Task of Interest at top, center of whiteboard. 


^Divide the space below into three columns; label as shown. 

* Elicit an example of one element of expertise, using the definitions and probes provided. Start with 
the first probe, (e.g., “Is there a tune when you walked into the middle of a situation and knew 
exactly how things got there and where they were headed?) 

* Elicit information for the remaining two columns before proceeding to another element: 

- Ask your SME, “In this situation, how would you know this? What cues and strategies are you relying 

on?" Record answers in middle column under “Cues and Strategies.” 

- Ask your SME, “In what way would this be difficult for a less-experienced person?" What makes it hard 

to do?” Record answers in final column under “Why Difficult?” 

* It is important that you cover the six basic Knowledge Audit probes; you may also want to use some 
or all of the optional probes. 


Task of Interest 

Example 

Cues 

& 

Strategies 

Why 

Difficult? 

Past & Future 
Example... 



Big Picture 
Example... 



Noticing 

Example... 




TIPS FOR DOING THE KNOWLEDGE AUDIT 

* Examples allow you to get at specifics and help you understand the task better. Ask for an example 

for each element of expertise. 

* Don’t try to write everything; but write enough so you will know later what was said and meant. 

With practice you will develop a sense of the level of detail you need. 

* Some of the questions may take a few minutes for the SME to answer thoughtfully; don’t rush; give 

the SME time to think over what you are asking about. 

* Confusion about what to write and in which columns may be a signal that the SME has 

misunderstood your question; the information you are getting is not what you expect. You may 
want to take a timeout, restate the question, and check that your SME understands what you are 
trying to get at. 
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ELICITING INFORMATION WITH THE KNOWLEDGE AUDIT 

Provide an explanation of the type of information you want; then ask the probe questions. You can 
read the definitions below or paraphrase them. 


BASIC PROBES: 

* Past & Future Experts can figure out how a situation developed, and they can think into 

the future to see where the situation is going. Among other things, this can 
allow experts to head off problems before they develop. 

Is there a time when you walked into the middle of a situation and knew exactly how 
things got there and where they were headed? 

* Big Picture Novices may only see bits and pieces. Experts are able to quickly build an 

understanding of the whole situation — the Big Picture view. This allows the expert 
to think about how different elements fit together and affect each other. 

Can you give me an example of what is important about the Big Picture for this task? 
What are the major elements you have to know and keep track of? 

* Noticing Experts are able to detect cues and see meaningful patterns that less-experienced 

personnel may miss altogether. 

Have you had experiences where part of a situation just “popped’ out at you; where you 
noticed things going on that others didn’t catch? What is an example? 

* Job Smarts Experts learn how to combine procedures and work in the most efficient way possible. 

They don’t cut corners, but they don’t waste tune and resources either. 

When you do this task, are there ways of working smart or accomplishing more with 
less — that you have found especially useful? 

* Opportunities Experts are comfortable improvising — seeing what will work in this particular 

situation; they are able to shift directions to take advantage of opportunities. 

Can you think of an example when you have improvised in this task or noticed an 
opportunity to do something better? 

* Self Experts are aware of their performance; they check how they are doing and 

make Monitoring adjustments. Experts notice when then performance is not 
what it should be (this could be due to stress, fatigue, high workload, etc.) and 
are able to adjust so that the job gets done. 

Can you think of a time when you realized that you would need to change the way you 
were performing in order to get the job done? 


OPTIONAL PROBES : 

* Anomalies Novices don’t know what is typical, so they have a hard tune identifying what is 

atypical. Experts can quickly spot unusual events and detect deviations. And, they 
are able to notice when something that ought to happen, doesn’t. 

Can you describe an instance when you spotted a deviation from the norm, or knew 
something was amiss? 

* Equipment Equipment can sometimes mislead. Novices usually believe whatever the 

* Difficulties equipment tells them; they don’t know when to be skeptical. 

Have there been times when the equipment pointed in one direction, but your own 
judgment told you to do something else? Or when you had to rely on experience to 
avoid being led astray by the equipment? 
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7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission.) 


ACTA - SIMULATION INTERVIEW 

Purpose: The Simulation Interview provides a view of the expert’s problem- solving processes in 
context. The interview provides specific detailed information on expert cognitive processes. 

How to get started: You will need to obtain a simulation of the task. The simulation you choose 
should address difficult, challenging elements of the job. It does not have to be high fidelity; it can be 
a paper and pencil simulation, a video depicting a scenario, or whatever is available. It is important 
that the simulation you choose presents a challenging scenario. 

CONDUCTING THE SIMULATION INTERVIEW 

• Divide a whiteboard into five columns, labeled as shown on the next page. 

• Have the SME experience (i.e. read, watch, interact with) the simulation. 

- Tell the SME, “ As you experience this simulation, imagine you are the ( Job you are investigating) in 
the incident. Afterwards, I am going to ask you a series of questions about how you would approach 
this situation.” 

• Elicit a list of the major events in the simulated incident and record in the first column. 

- Ask your SME, “Think back over the scenario. Please list the major events that occurred during the 
incident. These events could include judgments or decision points. As you name them, I am going to list 
them in the left column of the board.” 

• Begin with the first major event and elicit information for the remaining four columns before 

proceeding to the next major event. 

- Ask your SME, “As the ( Job you are investigating) in this scenario, what actions, if any, would you 
take at this point in time?” Record answers in the second column under Actions. 

- Ask your SME, “ What do you think is going on here? What is your assessment of the situation at this 
point in time?” Record answers in the third column under Situation Assessment. 

- Ask your SME, “What pieces of information led you to this situation assessment and these actions?” 
Record answers in the fourth column under Critical Cues. 

- Ask your SME, “What errors would an inexperienced person be likely to make in this situation?” 
Record answers in the fifth column under Potential Errors. 


TIPS FOR DOING THE SIMULATION INTERVIEW 

• Eliciting major events is critical to this interview. The major events should be turning points or 

segments of the story. You do NOT want a recount of the entire scenario. 

• People often want to critique the simulation. Assure your interviewee that you are interested in their 

critique, but that for the first part of the interview, you would like to work with the scenario as it 
has been presented. Be sure to follow up and ask for a critique at the end. 

• Don’t try to write everything; but write enough so you will know later what was said and meant. 

With practice you will develop a sense of the level of detail you need. 

• Confusion about what to write and in which columns may be a signal that the SME has 

misunderstood your question; the information you are getting is not what you expect. You may 
want to take a timeout, restate the question, and check that your SME understands what you are 
getting at. 
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7.1 Appendix A: ACTA Toolkit Methods (Copyright Klein Associates, Inc. Reprinted with permission. 


EXAMPLE: EXCERPT FROM A FIREGROUND SIMULATION 


Events 

Actions 

Situation Assessment 

Critical Cues 

Potential 

Errors 

On-scene 

arrival 

- Account for people 
(names) 

- Ask neighbors (but 
don’t take their 
word for it, check it 
out yourself) 

- Must knock on 
doors or knock it in 
to make sure 
people aren’t there 

- It’s a cold night, 
need to find place 
for people who have 
been evacuated 

- Night (time) 

- Cold — 15 degrees 

- Dead space 

- Add on floor 

- Poor materials 
(wood) punk board 
metal girders — 
buckle and break 
under fire 

- Common attic in 
whole building 

- Not 
keeping 
track of 
people 
(could be 
looking 
for people 
who are 
not there) 

Initial attack 

- Watch for signs of 

- Faulty construction: 

- Signs of building 

-Ventilating 


building collapse 

building may 

collapse include: 

the attic, 


- If signs of building 
collapse, evacuate 
and throw water on 
it from the outside 

collapse 

what walls are doing, 
cracks (building 
ready to collapse), 
floor groaas (floor 
ready to cave in), 
metal girders 
(click — coming out 
of wall — popping), 
cable in old buildings 
holds wall together, 
fire collapses walls 

this draws 
the fire up 
and spreads 
it through 
the pipes 
and 

electrical 

system 
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7.2 Appendix B: Preliminary Screening Questionnaire 


1. General Information 

Full Name: 

First, Middle, Last 

Address: 

Street and Number, or P.O. Box 


City, State, Zip Code, and Country (if not USA) 
Home Phone: ( ) Work Phone: ( ) 

Birth Date: 

Montli/Day/ Y ear 

Do you wear corrective lenses when you fly?(circle one) Yes No 


2. General Experience Information 

Current/Most Recent Airline: 

Current/Most Recent Position: 

Captain, First Officer, Engineer, etc. 

Are you currently flying military? (circle one) Yes No 

Years Flying Commercial (approximate): 

Years Flying Military (approximate): 

Total Hours Flying (approximate): 

Total Hours Flying as Pilot-in-Command (approximate): 

Years of formal education: (e.g., high school graduate =12) 
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7.2 Appendix B: Preliminary Screening Questionnaire 
3. Specific Aircraft Experience Information 


Please list the types of aircraft on which you have experience, beginning with the most recently flown. 

For each aircraft, please check the columns to indicate your approximate number of hours flying 
experience and approximate number of hours simulator experience. 

If you were an Instructor (I) or a Check Airman (CA) on any of these aircraft, please indicate by checking 
the last column. 


Aircraft Type 

Hours in Type 

Simulator Hours 

I/C A ? 


<300 

300-1000 

> 1000 

0 

<50 

>50 



































































Please check the appropriate column to indicate the approximate number of years of experience you have 
for each of the following categories: 


Specific Aeronautical Experience 

| Years Experience | 


< 1 

1-5 

>5 

Long-range, Over- water (Class II) Operations ( 2 engines) 




Long-range, Over- water (Class II) Operations ( > 2 engines) 




Total Multi-Engine (Captain or L/O, Military or Civil) 




Glass Cockpit (i.e. ELIS/CRT or FMS) 
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7.3 Appendix C: Phase 1 Interview Guide 


1. Inform the pilot that what they tell us will be confidential. Ask for permission to tape record 
the interview. (Take good notes!) 

2. State purpose of our research. Say something like: 

Becky, Rob, and Jason work for a small R & D company called Klein Associates. Klein 
Associates studies decision makers in many different domains in order to develop unproved 
training programs and to design decision support systems. Kara works for NASA Langley's 
AWIN (Aviation Weather Information) program. We want to learn how pilots use weather 
information when they make decisions before and during then flights. 

3. Obtain background information on pilot [review preliminary questionnaire from 
recruitment] . 

• Where, geographically, do you usually fly? 

• How varied are the routes you fly? 

• Are there any typical weather systems along your typical routes? 

• What degree of pressure do you feel to make a trip? 

• Under what FAR Parts do you operate? 

• What types of "cargo" do you carry? Do you have any sensitivity associated with this 
cargo? 

4. Focus the interview. We’ve read that weather is identified as the causal factor in 30% of all 
general aviation accidents. We have also read that most fatal weather-related accidents occur 
during the cruise phase of the flight. 

• Do you think these facts hold “true” for business jet operations? 

• Why? Why not? 

5. Use basic questions from the Task Diagram figure along with the list of the 12 phases of 
flight to identify the most cognitively challenging tasks for the pilot that weather conditions 
may impact. 

6. Use the Knowledge Audit to follow-up on the tasks identified with the Task Diagram. Try 
to elicit the pilot’s expertise for decision situations impacted by weather conditions. Try these 
probes: 

• Big picture 

• Noticing 

• Past & future 

• Opportunities/Improving 

• Anomalies 

• Equipment difficulties 

7. Obtain a list of all the weather products the pilot currently uses or has used in the recent past. 
You can use the Weather Products Summary sheet to help collect and/or record this 
information. 
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8. Collect an incident. Try a probe something like this: 

You can probably remember numerous flights in which you had to make difficult decisions 
due to the current weather conditions. We would like you to describe an incident in which your 
experience using weather products made a difference in how you handled the situation — a 
situation in which a less-experienced pilot might have made a different decision than you did. 
[Explore this incident for about 20-30 minutes, have some colored pencils & blank paper handy 
to get diagrams.] 

a. Construct the incident timeline. Be sure to relate this timeline to the different stages of 

flight operations. 

b. Review the timeline to verify the content and sequence of events. 

c. Identify key decision points involving weather. 

• Did you consider other alternatives? 

• Might someone else in this situation have responded differently? 

d. Situation Awareness probes: 

• What was it about this situation that let you know what was going to happen? 

• What led up to this decision? 

e. Cue probes: 

• What were you noticing at this point? 

• What were you seeing (or hearing) at this point? 

f. Knowledge probes: 

• What information did you use in making this decision? 

• How did you get that information? 

g. Goal probes: 

• What were your specific goals at this tune? 

• What were you hoping/intending to accomplish at this point? 
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Pilot 

Date 


Wx 

product 

What 

information 

does this 
product 
provide? How 
do you use this 
product? 

Where do 
you get this 
product/ 
When is it 
used (pre- 
flight, on 
flightdeck)? 

How 

credible/ 

trustworthy 

is the 

information 

provided? 

How 

current do 

you need 
this 

information 
to be? 

How 

important is 

this 

information? 
What impact 
does it have 
on your 
decisions? 

Additional 

comments/ 

notes 











7.5 Appendix E: Phase 2 Interview Guide 
Introductions 

• Background information on AWIN 

• Consent forms 

• Klein Associates Inc. 

• Schedule for the day, facility layout, overview of interviews 

Generating Cognitive Demands 

• “We want you to tell us about what you do when you fly a mission. We are interested in the 
judgments and decisions you make throughout your typical mission that are affected by weather. 
We have identified eight different phases of flight. For each phase we will ask you to describe 
the judgments and decisions you make that are directly affected by the weather.” 

• Use the phases of flight diagram to go through each of the steps listed on the following pages 
(e.g., “For the General Planning phase , what decisions/judgments do you make that are impacted 
by weather?”) Keep the participant focused on judgments and decisions that are related to 
weather. If they cannot generate their own cognitive demands, use prompts from the list. 

(Additional probes for eliciting cognitive demands for the different Phases of Flight ) 

• General Planning (24-48 hrs prior) 

What time should I depart/should my passengers be ready to depart? 

What are the general Wx problems for the day going to be? Factors to be aware of in Specific 
Planning? 

Is the destination feasible? Go/No Go? 

Is departure point feasible? 

Is there any reason why I shouldn’t fly at all? (Ex: menacing fog at departure or destination point) 

• Specific Planning (30 mins - 2 hrs prior) 

[ When is flight plan filed? How long does it take to put together a flight plan?] 

How long will the flight take? 

How much fuel will I need? (wind considerations) 

What’s my best route? (wind or other Wx problems) 

Are there any Wx obstructions? (fronts, t-storms) 

Is it safe to takeoff? If not, when? 

Will it be safe to land? 

Do I need to plan for alternates? Which alternates? 

Will I be able to get back in sufficient time? 

• Taxi/Take-Off 
IFR or VFR? 

Can pilot see runway? Runway contaminated? (ice, snow, foreign objects) 

Is there risk of icing? 

Are there dangerous wind conditions - downdrafts, windshear, etc? (aircraft configuration for 
taxi-ailerons) 

Request alternate runway to takeoff from? 

Are there aircraft configuration issues to think about for taxi due to wind? 

• Climb 

Is there risk of icing? 

Is aircraft performance affected by any current Wx condition? 
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• Cruise 

What's my destination like, will it still be safe to land? Is destination still viable? 

Will I need to divert to an alternate? Which alternate? (Holding and Wx problems) 

Is my route still appropriate (obstacles)? Change route? 

Is this altitude comfortable? (turbulence) 

Is my heading appropriate for navigation (wind)? 

Am I getting appropriate fuel consumption? (speed and altitude) 

Am I getting expected ground speed? Will I have enough fuel? 

Will I need to refuel on the way? If so, where? 

Is there risk of icing? 

Is there risk of lightening strike? 

Will I be there on time? Do I need to reschedule anything or communicate with work to 
reschedule meetings, etc.l 

Do I need to communicate with passengers and/or 3 ld parties about passenger schedules? 

Do I need to reschedule ground transportation at primary destination; arrange ground 
transportation at alternate airport ? 

• Descend 

Holding? Is it safe to enter holding pattern? Do I need to reschedule/coordinate with ground 
transportation? 

Are there weather phenomena that will affect aircraft performance (icing, severe winds, t-storms)? 
Is there severe windshear? Turbulence, microburst/downdraft activity? 

Is my destination still viable ? Do I need to divert to an alternate? 

Call FBO? Call tower ? 

• Approach/Land 

Am I going to land? 

Is my destination viable? [at this stage is the pilot already committed to land?] 

Do I need to go-around? Do I need to divert? 

Is the runway clear (visibility, ice, snow)? Request alternative runway due to Wx? (wind, runway 
obstructed) 

Is the ceiling OK? Is the visibility OK? 

Effects of cross winds? 

Are there downdrafts/microbursts? Windshear? 

Do I need to request an alternate runway? 

• Taxi/Deplane 

Obstructed runway? (ice, snow, other objects) 

Is the wind a problem (Passenger safety; am I going to get blown away; aircraft configuration)? 
Keep the plane inside or outside hanger? 

• Concurrent Tasks in all Phases of Flight 

[These issues may be present during all phases of flight, but are they always equally important 
considerations? Are there some times when these issues are more critical and therefore require 
more apparent decision support?] 

Is the Wx affecting my aircraft performance? What are my limits before I act, related to Wx ( e.g ., 
route, fuel, altitude, speed, schedule, safety, comfort, aircraft performance) 

Icing considerations? 
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Conduct the Simulation Interview using the ACTA Simulation Interview Guide 
Knowledge Audit Probes 

• Anomalies - New pilots don’t know what is typical, so they have a hard time identifying what is 
atypical. Experienced pilots can quickly spot unusual events and detect deviations. And they are 
able to notice when something that ought to happen, doesn’t. 

Can you describe an instance when you spotted a deviation from the norm (either in weather 
information on the flight deck or out the window), or knew something was amiss, that a new pilot 
wouldn’t notice? Example: After takeoff a pilot immediately noticed that the winds were not in 
the forecast direction on which his flight plan was based . . . 

• Equipment difficulties / unreliable information - Equipment and information sources can 
sometimes mislead. Information received from sources such as distanced operators is sometimes 
unreliable. New pilots often believe whatever the equipment or other information source tells 
them; they don’t know when to be skeptical. 

Have there been times when weather-related equipment ( weather information and/or weather- 
compensating equipment, e.g., deicing boots ) pointed in one direction, but your own judgment 
told you to do something else? Or when you had to rely on experience to avoid being led astray 
by weather information or weather-related equipment? Example: While enroute to his 
destination, a pilot became concerned about the reliability of the radar image he was getting on 
the flight deck . . . 

• Big Picture - New pilots may only see bits and pieces. Experienced pilots are able to quickly 
build an understanding of the whole situation - the big picture view. This allows the experienced 
pilot to think about how different elements fit together and affect each other. This can allow them 
to anticipate problems that might develop in the future. 

Can you give me an example of what the Big Picture looks like for this task? What are the major 
elements of the weather that you need to know about and keep track of ? How does this help you 
know where things are headed? Example: A pilot needed to fly around a hurricane to get to his 
destination, which had an uncertified AW OS operating. He had to rely on a variety of different 
sources of information to build his understanding of the weather situation including his past 
experience with similar airfields and other weather information. 

• Noticing - More experienced pilots are able to detect cues and see meaningful patterns that less- 
experienced pilots may miss altogether. 

Have you had experiences where you noticed things going on that indicated a potential weather 
problem that others didn’t catch? Example: When he was descending through a cloud, a pilot 
noticed a drop in temperature and the presence of precipitation. This pattern of cues alerted the 
pilot to potential icing conditions — normally he would expect temperature to increase as he 
descended. 

Elicit an incident using the Critical Decision Method 

“You can probably remember some flights in which you had to make difficult decisions due to the current 
weather conditions. We would like you to describe an incident in which your experience made a 
difference in how you handled the situation — a situation in which a less-experienced pilot might have 
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made a different decision than you did.” Note: If participant has difficulty recalling an appropriate 
incident — use the Scenario from Hell technique below. 

• Construct incident timeline, relating the timeline to stages of flight operations 

• Review the timeline to verify the content and sequence of events 

• Identify key decision points involving weather 

Did you consider other alternatives? 

Might someone with less experience have responded differently? 

To what degree was your decision based on past experiences, on formed training, on 
other’s experiences? 

• Situation Awareness probes 

What was it about this situation that let you know what was going to happen? 

What led up to this decision? 

• Cue probes 

What were you noticing at this point (visual-external, visual-instruments, auditory, 
aircraft response) 

Did you actively retrieve any additional information? 

Did you retrieve any additional information for the purposes of validating information 
you had? 

• Knowledge probes 

What information did you use in making this decision? 

Did you have any information that you didn’t trust enough to use it in making this 
decision? 

• Goal probes 

What were the specific goals you hoped to achieve by making this decision? 

What were you hoping to accomplish? 

Besides weather, what other factors were concerns? 

Elicit a "Scenario from Hell" 

“How would you design a weather-related “scenario from hell” for pilot simulation testing that would be 
difficult, but not impossible for another pilot. In particular, we are looking for weather-related events 
that are difficult to detect, ambiguous, insidious conditions, or conditions that, in conjunction with some 
other event or system failure, may result in an unsafe condition if not considered." 

• Review the incident to verify the content and sequence of events 

• Identify key decision points involving weather 

Might someone with less experience have responded differently? 

• Situation Awareness probes 

What about this situation that would let a pilot know what was going to happen? 

• Cue probes 

What would an experienced pilot have noticed at this point ( visual-external, visual- 
instruments, auditory, aircraft response) 

• Knowledge probes 

What information would an experienced pilot use in making this decision? 

• Goal probes 

What would the pilot be hoping to accomplish? 

• Besides weather, what other factors would an experienced pilot be concerned with? 

• Why would this situation be difficult for a less-experienced pilot? Where might they go wrong? 
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Mission : 

Current time is 4:00 pm. You have a pop-up trip taking the CEO and four other passengers to 
Jacksonville, FL. You arrange to depart from Norfolk airport at 5:30 pm. There is an evening meeting 10 
miles from the terminal at 8:00 pm that the passengers must make. You will also need to make a return 
trip the following morning to be back in Norfolk for a noon meeting. At the meeting, the CEO will 
discuss the bonus plan for the pilots and you must be there to collect. 

You are on the runway and making final preparations for take-off. Outside your window you see 
a windsock flying horizontal indicating that winds are gusting out of the Northeast at about thirty knots. 

It has been drizzling off and on during the day, however, you see no drizzle on the runway. Visibility is 
about seven statute miles and it is slightly overcast with a low ceiling at about one thousand feet. Current 
temperature is two degrees Celsius with a dew point of one degree Celsius. 

As you ascend to cruising altitude at flight level three five zero you're flying in and out of patchy 
clouds. The weather outside your window is clear now, but there are t-storm cells to the South forming 
large convective cumulonimbus clouds. The temperature outside is about negative forty-seven degrees 
Celsius. The flight is getting a little bumpy and passengers are beginning to make comments about the 
uncomfortable ride. 

You are listening to pilot frequencies to try to get a feel for what's ahead. You hear, "Charleston 
Northward at two two five zero, moderate turbulence at flight level three three zero, B757." 

Above flight level two four zero, you can see towering cumulous cells along your flight path 
Northwest of Wilmington, approximately 25nm South of Wilmington, a large system due East and 50nm 
from Charleston, and a smaller cell almost directly over Charleston International. 

Another pilot report indicates, "Dash Eight reported moderate mixed icing, forty miles East of 
Brunswick between six thousand and one zero thousand feet at two three one zero Zulu." 

When you start your descent towards Jacksonville, a pilot report conveys, "Moderate turbulence 
at two three two zero Zulu, on approach to runway 31 inside VOR at Jacksonville International, reported 
by DC8." 

Current conditions at Jacksonville indicate winds out of the Northwest at thirteen knots and 
gusting to twenty-one knots. There is a light rain and visibility is about two statute miles. Cloud ceiling 
is at one thousand two hundred feet. Current temperature is five degrees Celsius, with a dew point of four 
degrees Celsius. 

Materials Provided: 


Preflight 

Enroute 

Approach 

Charts 

Scenario Mission 

PIREPS - ride reports 

ATIS recordings 

Approach Charts 

Aircraft Description 

Visual descriptions 

PIREPS - terminal area 

Airport Diagrams 

Standard Briefing 
(DUATs encoded text) 
Weather Graphics: 
Current Surface 
24 Hour Surface 
48 Hour Surface 
Current Flight Rules 
Doppler Radar 
Lifted hidex 
Satellite Image 
Weather Hazards 

FW scripts 

METARS 

IFR enroute low alt. 

IFR high altitude 
Sectional charts 
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Dear Participant: 

This questionnaire complements the interview process in which you have just participated. Your 
responses on this questionnaire will help us interpret the results of your interview data and help us address 
some issues that we didn’t have time to cover in the interview process. 

The questionnaire has the following sections: 1) Characterizing Business Jet Operations, 2) Personal 
Limits, 3) Weather Flying & Aircraft Characteristics, 4) Weather Training, 5) Use of Weather Products, 
and 6) Weather Information Requirements for Business Jet Operations. The Characterizing Business Jet 
Operations section will help us get a better feeling for your operations and allow us to better interpret 
your interview results. In addition, it will help us define the special requirements and challenges of the 
business jet community. The Personal Limits section will help us understand how you might constrain 
how you fly as a function of weather and atmospheric conditions. The Weather Flying & Aircraft 
Characteristics section will help us understand how you use your knowledge of your aircrafts’ 
capabilities, and the equipage of your aircraft to make decisions in challenging weather situations. The 
Weather Training section will help us interpret your interview and questionnaire responses. In addition, 
we will better understand the breadth and depth of weather-specific training provided to business jet 
pilots. The Use of Weather Products section will help us understand the weather products (services, 
PIREPS, AIRMETS, graphics, . . .) you use, and the degree to which you find these products reliable and 
useful. Finally, the Weather Information Requirements for Business Jet Operations section will be used 
to complete a study of pilot weather information requirements by adding the needs of business jet pilots to 
those compiled for Commuter, General, and Transport pilot populations. 


Please complete this survey and mail it back to me in the attached self-addressed and postage-paid 
envelope. I hope to receive all responses back by February 29. If you are unable to return the survey 
before then, please simply call me and tell me when you think you will be able to return it. 

We are looking forward to developing prototype concepts for improving weather decision-making based 
on the interview and questionnaire results. We hope to see you again in the development and evaluation 
phases of these concepts. Thank you again for your participation in this study. Your time and efforts are 
much appreciated. 


Sincerely, 

Kara Latorella, Ph.D. 

Crew/Vehicle Integration Branch, AWIN Operator Support 

M/S 152 NASA Langley Research Center 

Hampton, VA 23681-0001 

k.a.latorella(a> larc.nasa.gov 

Phone: 757-864-2030, Fax: 757-864-7793 
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1. Characterizing Business Jet Operations 

1.1 Qwner/Qperator Characteristics: 

1.1.1 Size of Fleet : aircraft owned leased 

1.1.2 Frequency of fleet use: trips per 

1.1.3 Support structure (circle those you have in-house): 

dispatcher, initial training, recurrent training, maintenance, procedure/ checklist development 

1.1.4 Would you char acterize die aircraft you fly as: < 20,0001b 20, 000-40, 0001b > 40,0001b 

1.1.5 Do you share aircraft widi other organizations? Yes No 

1.1.6 Pilots are: Owners/Low Time Professional Crew 

1.1.7 FAR Operations (circle all diat apply): 

121 135 91 

1.1.8 Typical crew: pilots + non-piloting crew 

1.2 Pilot Perspective: 

1.2. 1 Advantages of being a business jet pilot : 


1 .2.2 Disadvantages of being a business jet pilot: 


1,2,3 Tasks you do beyond those of a commercial pilot: 


G-2 















7.7 Appendix G: Post-Experiment Questionnaire 


1.3 Mission Characteristics: 


1.3.1 Typical types of passengers: 


1.3.2 Typical reasons for flights: 

1.3.3 Amount of notice usually given for a flight: 

to (range), usually around 

1.3.4 How often are last minute changes to die trip? 

every time l/2time never 

1.3.5 Typical trip lengdi 

> 300nm, > lOOOnm, > 2500nm 

1.3.6 What % of your nips are to a novel destination? 

< 1% < 10% ~ 50% > 75% 

1.3.7 To how many different “typical” destinations do vou fly: 

1.3.8 Do you fly international trips? 

NO / YES: To where? 

1.3.9 Do you fly over water trips? 

NO / YES: How long over water? 


1.3.10 Add any goals that are missing from the precision of ET A at remote destination 

following list, and rank order these goals according to 

how important they are for your typical mission (i.e., precision of ETA on return to home base 

more important goals may be attained by sacrificing less 

important goals). fuel economy 

aircraft operating costs 

company transportation costs as a whole 

flexibility in mission 

passenger comfort 


(other) 
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1.4 Operational Elements 


1.4. 1 Factors you consider in planning a flight (especially any unique to business operations): 


1.4.2 Define a “good weather day.” 


1.4.3 What is die ratio of flights tiiat you fly in bad weatiier days (less dian “good weather days”). 


1.4.4 What would be “normal” bad weadier? (you’d go, you diink you’ll be fine, it’s just unpleasant). 


1.4.5 What would be “alerting” bad weadier? (you’d go, but diink you might be coming back). 


1.4.6 What would be the lower end of “prohibitive” bad weatiier (you wouldn’t go)? 


1.5 Problems 


1.5.1 What are the biggest problems associated witii business jet operations within your operational center? 


1.5.2 Please describe the biggest problems associated witii the aircraft you usually fly. 


1.5.3 What are the biggest problems of business jet operations as part of the NAS? 
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2.0 Personal Limits 


Personal limits are constraints on whether you fly, how you fly (i.e., manage the aircraft), as well as how 
you would perform other pilot duties during a mission (i.e., communications, systems management, task 
management). As such, when the conditions of a Personal Limit are met, a pilot will operate at a point 
inside the boundary established by “possible operations” (i.e., those permitted by FARS, those made 
possible by aircraft capabilities). Personal limits can be expressed in terms of IF/THEN rules: the “IF” 
describes the conditions under which you would exercise this limit, the “THEN” describes how you 
constrain your behavior because of this limit. Personal limits can be distinguished from “Organizational 
Limits.” A Personal Limit is one that you have determined through your own experience, by listening to 
others’ experiences, or that you have derived from your basic knowledge of weather and atmospheric 
phenomena, and aircraft performance characteristics. An Organizational Limit is one that has been 
provided to you through formal training, company procedures, aircraft pilot operating handbook. 

Please use the following table to describe the Personal and Organizational Limits that constrain your 
behavior beyond what is aerodynamically possible and permissible by regulatory agencies: 1) Indicate 
whether the limit is Personal (P) or Organizational (O), 2) Describe the task or decision that the limit 
affects, 3) Describe any Weather/ Atmospheric Conditions that pertain to this Limit, 4) Describe how this 
Limit alters whether or how you conduct a mission. 


P/O* 

Your Personal/ 
Organizational Limit (“IF”) 

Weather / Atmospheric 
Conditions 

If weather/atmosphere is worse than limit, 
what do you do? (“THEN”) 










3.0 Weather Flying & Aircraft Characteristics 

3.1 Please describe equipment (standard or special) that is on your aircraft that allows you to fly more safely in 
different weather conditions. 


Weather / Atmospheric Condition 

Aircraft Equipment 








3.2 Please describe how performance characteristics of the aircraft you usually fly affect how you fly in weather 
conditions. 


Weather / Atmospheric Condition 

Aircraft Performance Characteristic 
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4.0 Weather Training 

It is important for us to be able to interpret your responses during the interview and 
on this questionnaire in terms of your knowledge of weather flying. In the areas 
below, please list the formal classroom training, instructional videotape, computer 
training, and readings from which you have acquired skills and knowledge for flying 
in weather, for using weather services, and for using onboard weather-related 
equipment (in particular, onboard weather radar). 



classroom training 

videotapes 

computer-based 

training 

readings 

flying in weather 





using weather 
services 





using onboar d 
weather radar 





using other 

onboard 

equipment 






5.0 Your use of Weather Products 


The purpose of this questionnaire is to survey how biz-jet pilots use weather products (e.g., surface-level 
charts, the weather channel, internet) and weather resources to make decisions during flight missions. We 
want to know which products and resources biz-jet pilots use, where they get access to these products and 
resources, what information pilots look for in the products and resources they use, and in which stage of 
the mission the information is useful. The next page shows a table with six columns and multiple rows 
underneath. At the top of each column is a specific question related to specific weather products that you 
as a biz-jet pilot would use before or during a mission. These columns are: 

1. Wx product. This could be a specific product (e.g., pressure charts for the entire U.S.) or specific 
resources of where you get your weather products (e.g., a weather internet site, on-board radar, ATIS). 

2. What information does this product/resource provide? How do you use this product/resource? 

3. Where do you get this product (from the internet)? / When is it used (during pre-flight)? 

4. How credible/trustworthy is the information provided? 

5. How current do you need this information to be? 

6. How important is this information? What impact does it have on your decisions? 

Please start by identifying each weather product or resource you use during a typical mission, and then 
answer all subsequent questions about that specific product or source. We define a mission in the 
following order: General planning (24-48 hrs prior), Specific planning (l-2hrs prior), Taxi out and take 
off, Climb, Cruise, Descend, Approach and Landing, and finally Taxi in and deplane. We recognized that 
this may not be exactly what you define a mission to be, but would like you to focus on these stages. 
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Subject 2 - Cognitive Demand Table 
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7.8 Appendix H: Individual Cognitive Demands Tables 
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7.10 Appendix .1: Simulation Interview Summar 


SME 4 


Decision 


Planning Phase 


Route planning What is the best 
route? 


SME 5 


SME 6 


Will icing be a 
problem? 


Should I carry 
extra fuel? 


What are the 
significant 
weather 
concerns? 


Which runway 
should I use? 


Takeoff 

concerns 


Takeoff in 
cross winds? 


What are the 
cloud tops? 


What altitude 
should I fly at? 


T-storms 



Will icing be a 
problem? 


Should fuel load 
be heavy or 
light? 


Where are die 
wx concents? 


Which runway 
to takeoff from - 
cross winds are 
heavy 


Should I turn 
anti-ice on 
during climb and 
descent? When 
should I initiate 
anti-icing? 
Aircraft 
performance 
considerations. 


Will I need to 
carry extra fuel? 


What are major 

weadier 

concerns 

(destination, 

cruise and 

departure)? 

Icing? 


Should I request 
alternate runway 
on takeoff - 
cross winds; 


Do I fly into the 
wind or battle 
the crosswind? 


What is the best 
altitude to fly at? 


Am I within 
cross winds 
limit? Should I 
split controls 
with co-pilot — 
cross winds. 


What are the 
cloud tops? 


Should I request 
for higher 
altitude to get 
above 
turbulence? 


SME 7 


Where am I 
going? Domestic 
or international? 


Is diere going to 
be icing? 


SME 8 




Will icing be a 
problem? 


How much fuel 
should I carry? 


What are major 
wx concerns? 
Departure and 
destination 


Should I call 
Flight Service to 
speak with 
meteorologist? 





Should I ask 

How do 1 handle 

What are my 

tower about 

wx in the 

takeoff 

windshear? 

terminal? - 

concerns? 

Should I wait for 

gusty winds. 


windshear to 

visibility 


pass? — tends to 

Should I takeoff 


move quickly 

with anti-ice? 



How am I going 
to deal with 
gusty winds and 
cross winds? 




What are cloud 
tops? 




How am I going 

How am I going 

to avoid t-storm 

to avoid t- 

and turbulence? 

storms? 
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7.10 Appendix .1: Simulation Interview Summar 


Decision 


Turbulence 


SME 4 


SME 5 



Talk to ATC? 


Climb/ 

Cmise 


Turbulence 


Altitude 


Brief px 


Contact ATC 


SME 6 


How am I going 
to handle 
turbulence? 
Should I alert 
passengers about 
turbulence? 


SME 7 


Am I within 
personal limits? 




Where do I go if 
there is a 
problem in 
takeoff? Can I 
get back? 


Do I need to call 
ATC for flow 
control? What 
are other planes 
doing? Should I 
change vector? 



Which runway 

Should I make 

should I use at 

call ahead to 

JAX? 

change meeting 
times with 
clients? - may 
have to vector or 
hold 



How do I handle 
tur bulence and 
convective 
activity? 


Is this a good 
altitude? 


Do I need to 
brief passengers 
for rough ride? 


Where are the 
thunderclouds? 
How to avoid 
them? 


Do I need to call 
ATC to see what 
other aircraft 
have been 
doing? 


Do I need to 
listen to PEREPS 
— turbulence 
reports 


Should I request 
altitude change? 


Do I brief 
passengers/copil 
ot — windshear? 


How should I 
deviate around 
clouds? 


Should I call 
ATC for 
rerouting 
information? 


turbulence on 
climb out? 

Do I need to call 
ATC to see what 
other aircraft 
have been 
doing? 


clear-air 

turbulence? 


SME 8 


How can I make 
this flight more 
comfortable? 


How much do I 
brief my copilot? 


What do I do if 
an engine goes 
out? 


Will ATC allow 
me to do 
deviations I may 
want to do 
enroute? 


What is going on 
in JAX? What 
does the 
alternate look 
like? 

Should I call 
FSS for 
NOTAMS? - 
JAX and MCO 


How do I handle How do I handle 




Do I ask ATC 

What is the best 

for altitude 

altitude? Should 

change? 

I call ATC for 
change? 



What is the 
convective 
activity like? 


Should I talk to 
ATC for 
vector/heading 
change? Fly 
West? 


How do I avoid Can I pick 

storms? Ar e through 

there embedded weather? 
storms? 
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7.10 Appendix .1: Simulation Interview Summar 


Decision 


Fuel 


SME 4 


SME 5 



Should I turn 
anti-ice on early 
— no cost in 
performance in 
cruise and not 
much on descent 


SME 6 


What is my fuel 
state? Do I have 
enough for 
holding/divert? 


What are my de- 

icing/anti-icing 

capabilities? 



Approach/ 

Descent 


Which runway 



Which runway 
should I choose? 
What is runway 
length? 



Do I turn anti- ice 
on for approach/ 
descent? 




How do I pad 
my approach 
speed? 


How much 
approach speed 
should I carry? 



SME 7 


How’s my fuel? 


Is there icing? 


SME 8 



Will I need to 
anti-ice enroute? 
Will there be 
icing on 
approach? 


Should I fly 
faster? - Winds 
are heavy 


When should I 
retract the 
landing gear — 
let water runoff 
gear? 

Should I turn 
continuous 
ignition on? 


Is the runway 
contaminated? 

Is JAX plugged 
up? What’s 
going on for wx 
at JAX? 




What is the best 

Which runway 

runway for 

should I use? 

landing? Is the 


wind direction 


aligned with 


runway? 



De-ice wings 
and engine? 


Should I brief 
flight attendant 
about 

turbulence? 


How am I going 
to handle 
landing? — wet 
runways/ 
hydroplaning 


Should I talk to 
tower to find out 
what other 
planes have 
done? 



How’s the ride 
quality for 
passengers? 


How should I 
handle the 
landing? 


What have other 
planes done? 


Should 1 brief 
passengers? 
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Decision 
Visual vs 
instalment 


SME 4 


SME 6 


SME 7 


Check ATIS 
Destination wx 

Otlier 


SME 5 

Is this an 
instrument or 
visual approach? 

Should I get 
ATIS? 

What is the wx 
like ar ound 
destination? 
Should I hold at 
higher altitude to 
wait out 
turbulence? 


SME 8 

Will I need to 
shoot an ILS 
approach? Can 
field support it? 

What is ATIS 
reporting? 

Is my destination 

runway 

contaminated? 

Will I need to What’s my 

change flap Vref? What’s 

settings? my descent rate? 


Will I need to 
ask for different 
approach? 























7.11 Appendix K: Critical Decision Incident Tabies 

Incident 1 - Subject 2. 

SME 2 and his co-pilot were scheduled to fly into Elizabeth Town, NC from 

Norfolk, VA on a Cessna 525. They knew before takeoff that Elizabeth Critical cues : small, uncontrolled 

Town was a small, uncontrolled airport (sometimes they can’t even get an airport; non-precision approach; 500 
ATIS report from there) with a 500ft minimum. They also knew that the feet minimum ceiling 
approach into Elizabeth Town would be non-precision. 

The weather out die window was cloudy en route, hi fact, at a distance of 
about 200 miles out the pilots could not see Elizabeth Town, where on a 

clear day Elizabeth Town would have been visible. They listened to the Cognitive demand : What are the 

ATIS reports (a specific radio frequency for every airport) for weather weather problems at my destination? 

conditions at Fayetteville, an alternate airport in die Elizabeth Town area, Information sources : ATIS, pilot’s 

with die intentions of finding a better place to land. The weatiier at eyes, pilot’s experience with local 

Fayetteville reported marginal conditions so diey decided to check other weadier conditions 

local airports to compare weadier reports. Relying on his experience and Critical cues : cloudy, destination not 

understanding of weather and many years of flying, SME 2 also considered visible at 200miles; 

diat die weather in Elizabeth Town might not be as bad on die ground as it Cognitive demands : Is my 

looked from die sky, and may even have better weather dian die surrounding destination viable? How will I deal 

airports. The biggest concern for SME 2 was die overcast, cloudy with potential turbulence during 

conditions. Clouds usually have turbulence associated witii diem and diey descent? 

wanted to avoid diat. Actions : check weather at alternate 

destination 

SME 2 continued to listen to ATIS reports from die odier surrounding Why difficult? Weadier is dynamic 

airports as he approached Elizabedi Town. However, before a decision to and available weather information 
land was made SME 2 performed one more check. He called die FBO on die often is not current 
ground at Elizabedi Town to get real-time observer perspective. The FBO 
reported to SME 2 diat he was unable to see across die airport, which 

indicated to SME 2 diat he needed to find an alternate airport to land at. A Actions : continue to monitor ATIS 

call to die FBO on the ground is preferred over a call to the tower. A call to reports; call FBO at destination 

the tower usually indicates a strong commitment to land at diat specific Cues : FBO at destination couldn’t 

airport and if you are just checking the weather, diere is no need to call die see across ah' field; weatiier at 

tower. alternative destination better 

Information sources : destination 

SME 2 called ATC in Fayetteville and arranged to land there as diey were FBO; ATIS 
reporting better weather and were the closest alternative. He also arranged 
for ground transportation to pick his passengers up at the Fayetteville airport. 

Actions : arrange to land at alternate 
destination; arrange alternate ground 
transportation for passengers 
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7.11 Appendix K: Critical Decision Incident Tables 


Incident 2 - Subject 1 . 

SME 1 was scheduled to fly into the St. Simons, Georgia airfield, a small Cognitive demand: What’s the 
island off the coast of Georgia, out of Norfolk, VA. SME 1 waited until the general weather picture? What are 

day of the mission to make his final decision about flying into St. Simons the weather problems for this flight? 

because of a hurricane watch in the area. He thought that conditions might 

change overnight. He knew that one trip in particular, out of the three he had Critical cues : hurricane watch for 
to make to St. Simons that day, was going to be difficult because of the destination 

timing and proximity of the hurricane. He used a one-page FAA briefing and 

the internet to assimilate his forecasted weather data. Information sources : FAA briefing; 

internet 

This was the first trip into St. Simons but was confident with his knowledge 

of local weather conditions. SME 1 also had experience flying into other Information sources : pilot’s 
airfields (Hilton Head and Brunswick) that are similar to St. Simons, and in experience with local weather 
similar weather conditions. SME 1 knew that die AWOS at St. Simons was conditions; AWOS report from 
uncertified in handling weather reports. SME 1 also had alternative airports destination 

in mind (Savannah, Brunswick) before leaving. Why difficult? AWOS at destination 

known to be uncertified; threatening 
weather at destination 
Cognitive demand : Is my destination 
viable? 

Before taking off, SME 1 set more restrictive limits on wind speed and Actions : consider alternative 

direction than he usually does. If winds surpassed die limits SME 1 set, he destinations 

would immediately reroute to an alternative airport, hi fact, in some Cognitive demand: How will I deal 

airplanes SME 1 would not have even attempted to make diis trip. He with weather obstacles on descent? 

contacted die FBO at St. Simons to get a real-time assessment of die Actions : set more restrictive limits 

situation on die ground, and received near real-time data from Brunswick (15 on permissible wind speed and 
minutes old). direction for landing; contact FBO at 

airport near destination airport; 
listen for weather updates for otiier 
nearby locations 

On final approach, SME 1 prepared himself by using Hilton Head as a Cues: wind speed and direction at 

template for landing on St. Simons. The features were virtually the same: a destination 

small island, the climate was similar, and SME 1 knew it had to be a Information sources : FBO at airport 

precision approach. He landed with no problems. near destination 

Cognitive demand : What kind of 
landing do I need to make? 

Why difficult : he'd never landed at 
this airport before 
Information sources : pilots’ 
knowledge of other similar airports 


K-2 













7.11 Appendix K: Critical Decision Incident Tables 


Incident 3 - Subject 6. 


SME 6 was scheduled to fly to London. Ontario out of St. Paul at 5:00pm in 
a Gulf Stream 4. Total flight tune is 90 minutes, about 450-500 miles. From 
about 4:00 — 4:45, SME 6 gathered pre-flight weather information for 
departure and destination airports. He primarily uses KAVOURAS, a 
weather service based hr St. Paul. Tire service provides verbal briefs, 
including the ability to file a flight plan, it provides a radar picture, and 
provides weather for departure city and state. Tire only problem with this 
service is that it doesn't provide tire big picture. After collecting initial 
weather reports, SME 6 had a discussion with tire 1 st officer about the trip in 
which the 1 st officer voiced his concern and drought drey shouldn’t go. SME 
6 felt otirerwise, and asked die 1 st officer to call dre ATC tower in London. 
The conditions at St. Paul were rainy, foggy, below minimums, and 
temperatures were about 3 to 4 degrees Celsius. London was reporting 
conditions below minimums but drey had unproved to 100 ft and !4 mile 
visibility. They were also reporting dre cloud bottoms were ragged. SME 6 
knew that even diough conditions might be below minimums when diey 
arrived in London, he could at least go look ( FAA Part 91 regulations). 

Before diey left diey called dre tower hi London once agahi, which reported 
conditions were down again, 200 ft and V 2 mile visibility. They decided to 
go anyway and briefed die passengers on die situation. They devised several 
alternatives if they couldn’t land hi London and agreed diat die best 
alternative would be to turn around and go back to St. Paul. Detroit could 
have been an option but required more logistical planning. 

Because the reports out of London were “ragged,” diis meant conditions 
were up and down all day in SME 6’s mind. From his previous flying 
experience, SME 6 knew that minimums at night were different dian during 
the day. At night pilots can see the amount of light sliming up through die 
clouds from the landing strip. Obviously, day conditions don’t provide tiiis 
cue. SME 6 knew he could always go back to St. Paul if the weather was 
bad, but he wanted to take a look anyway. 


At about 6:00pm, durhig cruise, SME 6 checked with FW. They had enough 
fuel to play with several options: hold for a while, make a pass, or go back to 
St. Paul. On Approach/Descent, SME 6 briefed 1 st officer (check 
frequencies, radios, minimums check, and initiated call-outs). They 
contacted ATC. ATC reported that other planes had landed that evening, 
which indicated to SME 6 that he would at least try to land. Minimums were 
300 — 400 ft, and 1 mile visibility. SME 6 inferred this from the amount of 
light shining up through the clouds. 

They were able to land without any problems 


Cognitive demands : What’s the 
general weather picture? What are 
the weather problems at destination? 
Why difficult? 1 st officer disagreed 
with pilot about viability of landing 
Information sources : private weather 
service; ATC at destination tower 
Cues : radar picture; temperature, 
ceiling, and visibility at destination 
and departure airports 
Action : discuss destination weather 
with 1 st officer; call ATC at 
destination tower; plan to fly to 
destination and then make decision 
about landing 

Actions : call ATC at destination 
tower (again) 

Cues : destination ceiling/visibility 
Actions : brief passengers; identify 
alternate destination which was to 
return to departure airport if unable 
to land 

Cognitive demand : Can I anticipate 
and deal with significant weather at 
destination? 

Why difficult? Ceiling and visibility 
conditions at destination kept 
changing throughout the day and 
evening 

Cues: landing lights shining through 
low level clouds 

Action : fly to destination air field 
and look at lights 

Cognitive demands : Is my 
destination still viable? How am I 
doing on fuel? 

Action : contact ATC at destination 
tower 

Cues : ATC reported other planes 
had landed recently 
Action : attempt landing 
Cues : light shining through low 
level clouds 

Information sources: ATC at 


destination tower; pilot’s eyes 









7.11 Appendix K: Critical Decision Incident Tables 

Incident 4 - Subject 5. 

SME 5 and his co-pilot were scheduled to fly a group of Asian business- Cognitive demands: What’s the 

people from an airfield in Phoenix, AZ (Grand Canyon) to Morristown, NJ, general weather picture? Are there 

in a Gulf Stream 4 at 4 pm (6pm EST). SME 5 was planning to stay over weather problems at destination? 

night in NJ, and then fly back to MN in the afternoon. At about 2 pm (4 pm Why difficult? Weather is dynamic 

EST) SME 5 and co-pilot began planning for die trip to die East Coast. and available information often is 

Conditions on die ground hi Phoenix were clear and beautiful. He called die not current 
Wx service before diey left and diey said diere were no problems in die NY Cues : gusty winds at destination 
area. Current conditions hi NJ at around 2pm were reporting marginal widi Information Source : private weather 
some gusty winds. service provider 

At about 8:30 EST SME 5 started his approach hito Morristown. Prior to Cognitive demands : How do I deal 
diis point die ride had gone smoothly and diey didn’t expect any problems with significant weadier at 
ahead. About 100-150 miles out SME 5 picked up die Morristown ATIS, destination (gusty winds)? Which 
which reported winds gusthig from die NW at 15-20 knots. They were runway should I use? 

approaching from die Soudiwest and he concluded diat runway 5, landing to Cues : winds speed and direction, 
die Nordieast, was die only choice. The Morristown airport sits down hi a runway direction, topography 

valley surrounded by some terraced hills on die Soudiwest side and a swamp surrounding destination airport 

area to die East. He knew diat diere would be a significant crosswind. He Information sources : ATIS, pilot’s 

was not dimkhig about alternatives because the weather was not diat bad. familiarity with runways and 

topography at destination airport 

He started to descend to about 20,000 ft when he received notice from ATC 

diat diere was turbulence below 10-15,000. At almost 9pm EST, he could Cognitive demands : How do I deal 
see Morristown 20 miles out. SME 5 was cleared for a visual approach. with significant weadier at 

When he got down to 6000 ft die winds were worse there tiian diey were destination (gusty winds)? What 

higher up. He hadn’t expected diis. He got down to 4000 feet and it was khid of approach do I need to make? 

quite rough. At about six miles out from die airport diey started to get Information sources : ATIS, pilot’s 

banged around a lot. The passengers were no longer talking. He had the visual & kinesthetic experience 

aircraft on autopilot and auto dirottle, but it was not handling die winds well. Cues : turbulence reported, cleared 

so he turned it off around 2000 ft. This caused a warning horn to sound. The for visual approach, experienced 
passengers heard this and were very alarmed. high winds at lower altitude, 

turbulence experienced 

As he was coming over the hills the winds stalled getting violent, diere were Actions : turn off autopilot&/tiirottle. 
heavy gusts up and down, rolling and yawing. The normal approach speed Cognitive demand: What kind of 

was 130- 140 knots, but they were flying 160- 170 knots. His yoke control approach do I need to make? What 

was going from stop to stop (/. e . , he was losing control of the aircraft). They kind of landing do I need to make? 

had to abort the approach. He knew he could get the aircraft to the runway Why difficult? Unexpectedly high 

but he didn’t think he could keep the plane on the runway when they landed. winds made aircraft control difficult 

Cues: yoke control handing difficult 

It was very unusual to have these violent gusts. The plane didn’t feel right. Action : Abort the approach 

SME 5 now knew that runway 5 wasn’t going to work so he looked at Cognitive demand : should I try a 

runway 30. This was too short and he was too heavy to land there. He different runway? 

decided to abort the landing. Cues : High winds, runway too short, 

plane too heavy 

They went back up and circled for 15-20 minutes so he could calm down and 

discuss his options with his 1 st officer. All the NY airports were gusty, but Cognitive demands : Is my 

they didn’t all have the hills like Morristown. They checked with destination still viable? How am I 

LaGuardia, but they were real busy. At around 9:30pm EST they tried to doing on fuel? 

land again at Morristown. They had enough fuel to try again and if they Action : discuss options with copilot 

couldn’t land it this tune they could still go somewhere else. Information sources : checked with 

ATC at alternate airport 

He did not get any new information on the winds. No one else had landed at Cues : alternate airport very busy, 
Morristown since he had tried 15-20 minutes ago (but no one had attempted sufficient fuel to try again 
it either). The turbulence was much less on the second try and they were Action : try landing at original 
able to land. destination (again). 
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7.11 Appendix K: Critical Decision Incident Tables 

Incident 5 - Subject 5. 

SME 5 was waiting oil the passengers who were late. His copilot was the 
most experienced pilot in the organization. The weather was beginning to 
turn nasty with a storm moving rapidly towar ds the field. They followed the 
weather on the internet and television in the crew room, and were getting 
updates from the tower for over an hour. The storm was moving in at about 
30 knots and would hit the field soon, at least delaying their departure 
further. 

The passengers arrived and boar ded the aircraft. The storm was moving in 
fast but current conditions at the field were good. No rain, good visibility, 
and the wind was not going to be a problem. They knew they would have to 
get up quickly and make a quick turn to avoid the storm. There was still a 
question of whether they would get off the ground in time, but ATC 
confirmed that they looked OK if they expedited their departure. They used 
their on-board radar to check die progress of die storm. They swung die 
aircraft 360 degrees on die ground to make sure diat die weatiier was clear 
around die storm and diat diey could get out. The radar picture was helpful, 
aldiough ground clutter made die picture a little ambiguous. However, die 
tower’s radar picture was clearer, and the tower made die same assessment of 
die situation. The speed and intensity of the storm had worried die pilot. 

They had watched it move rapidly hi dieir direction, and dieir departure was 
hi jeopardy. Thirty miles away a tower had been abandoned due to die 
storm, so they knew it was severe. Both die pilot and die 1 st officer agreed 
diat they could make it, and dieir assessment matched die tower’s 
assessment. They knew it was heading right for diem because diey could see 
from die weadier reports and tracked its movement, and as diey prepared for 
departure, diey could see die growing, ominous clouds. The pilot was 
confident hi die judgment of his 1 st officer, an experienced colleague. Also, 
their assessment had been confir med by the tower. If either of these other- 
opinions had differed from the pilot’s, he would have postponed the 
departure. If the 1 st officer had been less experienced, the pilot may not have 
been so confident hi trying to beat the storm. 

They took off, deviated immediately, and made it clear of the storm. Two 
mhiutes later the storm hit the field (accordhig to reports from the tower at 
die field). 


Cognitive demands : What’s die 
general weather picture? What are 
die weather problems for departure? 
Information sources: internet, TV 
Cues : direction and speed of storm 
moving towards departure airport 

Cognitive demands : do I need to 
delay my take-off? How am I going 
to avoid weather obstacles durhig 
climb? 

Cues : lack of wind and precipitation 
at departure airport; visibility at 
departure airport, pilot observations 
Action : track approaching storm on 
on-board radar; talk to tower to get 
dieir assessment of approaching 
storm 

Information sources : ATC at 
departure tower, on-board radar 
Cues : radar picture of approaching 
storm, speed and intensity of 
approaching storm, local tower 
abandoned 

Action : discuss options witii 1 st 
officer 

Cues: experience of 1st officer; 
agreement with destination tower 
ATC 

Action : take-off quickly before 
storm arrives 

Action : turn aircraft immediately 
after take-off to avoid storm 
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7.11 Appendix K: Critical Decision Incident Tables 

Incident 6 - Subject 4. 

SME 4 had to miss an approach into a small, unsupported airfield near Cognitive demand : Can I deal with 

Traverse City, MI. He could not see the airfield even though the ASOS was low ceiling and limited visibility at 

telling him that the airfield was above minimums. SME 4 knew that the destination? 

temperature and dew-point spread was close, and that die wind was in die Information sources : ASOS at 

“wrong direction.’’ But die automated weather observation equipment was destination; pilot’s observation of 

telling him diat die airfield was widiin die limits. He said he “smelled a rat.” conditions at destination, pilots’ 

He made die approach, but die visibility was only 2 V 2 miles and die ceiling experience with similar weather 

was under 700 feet. He could not see die runway, so he made a missed situations 

approach. He did not really know how the automated observation equipment Cues: temperature/dewpoint spread; 
worked, but he sensed diat it might be wrong based on his assessment of die wind direction 
conditions, and his intuition proved him correct. He ended up flying to a Actions : attempt approach, observe 
close alternate airfield. ceiling and visibility 

Cue : couldn’t see runway 
Action : abort approach, fly to 
alternate destination 
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7.12 Appendix L: "Scenarios from Hell" Incidents 
Subject 3 - 

• Windshear at takeoff or landing 

• Add to workload at landing by changing runways (need to redo approach) 

• Low ceilings encountered just after decision height (expert will prepare for missed approach) 

• Forecast a close dew pt/temp spread 

• Runway clear until rotate speed reached, then socked in 

• Start flare on landing and runway in fog 

• While in holding pattern, ice accumulates on die bottom (of wings?) 

• Ice in clouds 

Any slow forming icing conditions (when flying slow, low altitude) [this is particularly difficult when on 
autopilot diat continues to adjust for it until overloaded] 

Subject 5 - 

• Strong wind sheer, unexpected at takeoff or landing 

• Engine failure on opposite side of cross-wind 

• Icing during night flight 

• Equipment failur es diat affect de-icing equipment (asymmetrical malfunctions are hard to control) 

• Contaminated runways 

• Pitot-static system failure widi slow onset (altitude and airspeed indicators are affected) 

• Radar clutter or misleading radar image (big cell behind die squall line) 

Subject 6 - 

• Destination is an uncontrolled airport, maybe no FBO 

• Snow or frost on runway — no braking action 

• Ceiling and visibility at minimums 

• Forecast winds with gust from 10-20 knots 

• Landing on runway at 3 degrees 

• EICAS gives message for anti-skid failure after takeoff, but then this goes away 

• EICAS message comes back on when pilot hies to put wheels down for landing [lie should abort landing and look 
for alternate with dry runway] 

• If fail to gear down, must fly at low altitude to alternate or red act them and do a belly landing 

• When aircraft starts to descend the winds are up to 25-30 knots with a 90 degree crosswind 

• Runway has blowing snow and anti-skid has failed 

• Takeoff from south 

• Have thunderstorms in the area 

• Running low on fuel 

• Runway 4500-5000 ft 

Subject 8 - 

• Forecast icing conditions (especially for night flights) 

• Use a plane with no deicing equipment 

• Forecast low visibility at destination 

• Co-pilot that hinders (doesn’t call out altitude or descent rate on approach) 

• Wake turbulence can be tricky 

• Strong wind sheer 

• Strong cross winds at takeoff and landing 

• Engine failure at takeoff with strong cross wind (makes physical control difficult) 

• Bleed air failure (system that affects deicing equipment) 

• Pitot-static system icing failure — slowly degrades validity in instruments 

Subjects lj2,4 and 7 did not receive this probe due to lack of time mailable in the interview. 
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7.13 Appendix M: Scenario Elements and Behavioral Markers. 



Behavioral 

Markers 


Weather on the 
ground may be better 
than weather in the 
air. Need to check 
with alternate 
airports if weather in 
surrounding area is 
better or worse 

Find smoother air - 
need to consider how 
this will affect 
performance of plane 
(higher vs. lower 
altitude, fuel 
consumption) 


Need to consider 
how this may impact 
approach with 
precipitation in the 
area (how long they 
will have to fly 
through potential 
freezing rain) 

May be ground 
clutter, need to get 
multiple reports 


Information 

Sources 

Airport directory; 
pilot's own 
experience 

Visual observation 
out the window, 
usually able to see 
from this distance; 
ATIS; PIREPS; 
ATC; FBO on 
ground 

Sensing the plane 
being jerked around 

FBO 

ATIS 

On-board radar 

Visual observation; 
ATIS, ATC 

Features Involved 

Infrastructu 

re 

X 







Environme 

nt 








Syste 

m 






X 


Weath 

er 


X 

X 

X 

X 

X 

X 

Phases of Flight 

< == 
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7.13 Appendix M: Scenario Elements and Behavioral Markers. 
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Behavioral 

Markers 

Pilot must consider 
how terrain will 
affect aircraft 
performance, and 
how weather may be 
different in valley 
than at higher 
altitudes. Also 
surrounded by hills - 
less room for error. 
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alternate landing if 
cannot land at single 
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Stream 4 
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7.13 Appendix M: Scenario Elements and Behavioral Markers. 



Behavioral 

Markers 

Over-reliance on 
weather equipment 
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Behavioral 

Markers 

Harder to look out 
the window and see 
ice build up, need to 
be more vigilant of 
other indicators e.g. 
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ance/ handling, air 
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anticipate the 
potential for icing 
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